Inducing Task Models from Computer-Use Traces

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
• Venue: arXiv: cs.CL
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20319v1
• PDF: https://arxiv.org/pdf/2608.20319v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have developed a method called Task Model Induction (TMI) to derive structured task models from naturalistic computer-use traces, which are records of screenshots and mouse or keyboard actions. These models can help computer agents understand how tasks are performed in real work environments and allow organizations to audit and reuse this knowledge. TMI is designed to handle multi-threaded activities with interleaved goals, a challenge for existing methods that assume a given task or single workflow. In controlled experiments, TMI successfully recovered interleaved tasks with high accuracy and improved task accuracy by 30% over the strongest baseline.

What the Document Says

• The research introduces a method called Task Model Induction (TMI).
• TMI is designed to derive structured task models from naturalistic computer-use traces.
• Task models are useful for computer agents in real work environments and organizations for auditing and reusing knowledge.
• Existing methods for inducing task models assume a given task or single workflow and produce step-level summaries rather than structured task models.
• TMI can disentangle concurrent activity and pair hierarchical objective models with control flow procedure models.
• In controlled experiments, TMI recovered interleaved tasks with 0.974 agreement against ground-truth groupings.
• TMI reconstructed 74.9% of the observed execution steps in these experiments.

Key Concepts Mentioned

• Task Model Induction (TMI)
• Naturalistic computer-use traces
• Computer agents
• Real work environment
• Organizations
• Hierarchical objective models
• Control flow procedure models
• Interleaved tasks

Claims Made by the Document

• TMI can derive structured task models.
• TMI is effective in handling multi-threaded activities with interleaved goals.

Evidence Used

• Recovered interleaved tasks with 0.974 agreement against ground-truth groupings
• Reconstructed 74.9% of the observed execution steps

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.