Inducing Task Models from Computer-Use Traces

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
• Venue: arXiv: cs.CL
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20319v1
• PDF: https://arxiv.org/pdf/2608.20319v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers present a new method called Task Model Induction (TMI) to extract detailed and structured task models from computer use traces, such as screenshots and mouse or keyboard actions. These task models can help computer-use agents understand how tasks are performed in real work environments and organizations to audit and reuse this knowledge.

What the Document Says

• Computer-use traces are valuable for deriving task models
• Task models matter for computer-use agents and organizations
• Inducing such task models is challenging due to multi-threaded activity
• Existing methods assume a given task or single workflow
• TMI introduces a method that disentangles concurrent activity and induces hierarchical objective and control flow models for each latent task
• TMI recovers interleaved tasks with high agreement against ground-truth groupings on controlled human and agent trajectories
• Skills derived from TMI's task models improve held-out task accuracy

Key Concepts Mentioned

• computer-use traces
• task models
• real-world work
• multi-threaded activity
• Task Model Induction (TMI)

Claims Made by the Document

• Computer-use traces are a valuable resource for deriving task models.
• Such task models matter in real work and organizations.
• Existing methods assume a given task or single workflow.
• TMI can disentangle concurrent activity and induce hierarchical objective and control flow models.
• TMI recovers interleaved tasks with high agreement on controlled human and agent trajectories.
• Skills derived from TMI's task models improve held-out task accuracy.

Evidence Used

• Observed in the text: 'Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions...'
• 'Such models matter as computer-use agents enter real work...', 'However, inducing such task models is challenging...', 'We introduce Task Model Induction (TMI), which...', 'Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings...', 'Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0%...'

Methods / Approach

• Task Model Induction (TMI)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The text does not specify the specific challenge in inducing such task models.
• The ground-truth groupings and strong workflow induction baseline are not defined or explained.

Open Questions

• What exactly are controlled human and agent trajectories?
• What is the strong workflow induction baseline mentioned in the text?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text does not provide specific details about the ground-truth groupings or the strong workflow induction baseline used for comparison with TMI.

Raw Abstract / Excerpt

Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.