ContextWeave: A Real-World Workflow Benchmark

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Bo Wang, Yuqian Yao, Enxi Wang, Luozhijie Jin, Yang Liu, Yiran Suo, Yuxuan Cai, Enyu Zhou, Yufei Gao, Honglin Guo, Tianyu Huai, Li Ji, Zhikai Lei, Bufan Li, Lizhi Lin, Jinxiu Liu, Jie Yang, Jiazheng Zhou, Maosen Zhou, Pengfang Qian, Shichun Liu, Guanshan Liu, Hao Zheng, Yunhao Yu, Hang Yan, Jihua Kang, Xinchi Chen, Xipeng Qiu
• Venue: arXiv: cs.AI
• Published: 2026-08-05
• Collected: 2026-08-06
• URL: http://arxiv.org/abs/2608.04830v1
• PDF: https://arxiv.org/pdf/2608.04830v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces ContextWeave, a new benchmark for evaluating the performance of language agents in long-horizon, stateful workflows. The benchmark focuses on memory as an essential component and measures workspace quality, alignment with user preferences, relevance, continuity, solvability, and robustness to misleading recall. The study finds that actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries.

What the Document Says

• The paper presents ContextWeave, a benchmark for evaluating the performance of language agents in long-horizon, stateful workflows.
• ContextWeave measures workspace quality and alignment with participant-specific preferences.
• It assesses diagnostics of relevance, continuity, solvability, and robustness to misleading recall.
• The study involves six memory components and five base models.
• Actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries.
• Misleading recall can also be more susceptible with actionable, experience-rich memory.

Key Concepts Mentioned

• ContextWeave
• language agents
• long-horizon workflows
• memory
• workspace quality
• user preferences
• relevance
• continuity
• solvability
• robustness
• misleading recall
• actionable memory
• compact summaries

Claims Made by the Document

• Actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries.
• Misleading recall can be more susceptible with actionable, experience-rich memory.

Evidence Used

• The text describes the findings of the study which show the differences in performance between actionable, experience-rich memory and compact summaries.

Methods / Approach

• The paper presents a benchmark (ContextWeave) for evaluating language agents' performance in long-horizon workflows.

Limitations Stated by the Authors

The paper does not provide details on the specific methods used for measuring workspace quality, alignment with user preferences, relevance, continuity, solvability, and robustness to misleading recall.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The analysis might be limited by the assumptions made by the authors about the behavior of language agents and memory systems.

Open Questions

• What specific methods were used for measuring workspace quality, alignment with user preferences, relevance, continuity, solvability, and robustness to misleading recall?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not provide details on the specific models tested or the exact metrics used in the evaluation.

Raw Abstract / Excerpt

Memory is essential as language agents move from isolated tasks to long-horizon, stateful workflows, yet existing evaluations often reduce it to retrieval or question answering. We introduce ContextWeave, a longitudinal benchmark that evaluates whether recalled experience improves downstream agent performance in realistic office-work streams. ContextWeave reconstructs privacy-preserved, multi-month workflows of 14 participants into 1,005 executable tasks, including 568 core evaluation tasks, with instructions, containerized environments, trajectories, and task-specific rubrics. It measures workspace quality and alignment with participant-specific preferences, complemented by diagnostics of relevance, continuity, solvability, and robustness to misleading recall. Across six memory components under a fixed model, the strongest configuration raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60. With a fixed memory component, recall improves both outcomes for all five tested base models, although gains vary substantially. Our analysis shows that actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries, while it can also be more susceptible to misleading recall. These findings motivate memory systems that optimize not only retrieval relevance but also reliable use during execution.