Inducing Task Models from Computer-Use Traces

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
• Venue: arXiv: cs.CL
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20319v1
• PDF: https://arxiv.org/pdf/2608.20319v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers present Task Model Induction (TMI), a method that extracts structured task models from naturalistic computer-use traces. These traces are screenshots and mouse or keyboard actions recorded during work. TMI aims to help computer agents understand how tasks are performed in real-world situations, and organizations reuse this knowledge. The method can disentangle concurrent activities, find the latent tasks within an unconstrained trace, and induce a hierarchical objective model and a control flow procedure for each latent task.

What the Document Says

• The research introduces Task Model Induction (TMI) for deriving structured task models from naturalistic computer-use traces.
• These traces consist of screenshots and mouse or keyboard actions recorded during work.
• TMI aims to help computer agents understand real-world tasks, assist organizations in auditing and reusing this knowledge.
• TMI can disentangle concurrent activities and discover latent tasks within unconstrained traces.
• For each latent task, TMI induces a hierarchical objective model and a control flow procedure.

Key Concepts Mentioned

• Task Model Induction (TMI)
• Naturalistic computer-use traces
• Computer agents
• Real-world tasks
• Hierarchical objective model
• Control flow procedure

Claims Made by the Document

• TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings on controlled human and agent trajectories.
• TMI reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline.

Evidence Used

• Intrinsic evaluation shows that TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps.
• Extrinsic evaluation shows that skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.

Methods / Approach

• Task Model Induction (TMI)

Limitations Stated by the Authors

• UNKNOWN (The paper does not mention any specific limitations attributed to the authors.)

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The agent's limitations may include interpreting the accuracy of the results based on the provided data and understanding the implications of the improvements in task accuracy.

Open Questions

• What are the potential applications of TMI outside of computer agents and real-world tasks?
• How generalizable is TMI across different types of work and computer use?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The research focuses on deriving structured task models for computer agents in real-world situations, but it may have broader applications or implications.

Raw Abstract / Excerpt

Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.