Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

ContextWeave: A Real-World Workflow Benchmark

Source

arXiv

arxiv.org/abs/2608.04830v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Bo Wang, Yuqian Yao, Enxi Wang, Luozhijie Jin, Yang Liu, Yiran Suo, Yuxuan Cai, Enyu Zhou, Yufei Gao, Honglin Guo, Tianyu Huai, Li Ji, Zhikai Lei, Bufan Li, Lizhi Lin, Jinxiu Liu, Jie Yang, Jiazheng Zhou, Maosen Zhou, Pengfang Qian, Shichun Liu, Guanshan Liu, Hao Zheng, Yunhao Yu, Hang Yan, Jihua Kang, Xinchi Chen, Xipeng Qiu
  • Venue: arXiv: cs.AI
  • Published: 2026-08-05
  • Collected: 2026-08-06
  • URL: http://arxiv.org/abs/2608.04830v1
  • PDF: https://arxiv.org/pdf/2608.04830v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers introduce ContextWeave, a new benchmark for evaluating the performance of language agents in long-horizon, stateful workflows. The benchmark measures how well an agent can recall and apply past experiences to improve its performance in various tasks. The study found that using actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries, but can also be more susceptible to misleading recall.

What the Document Says

  • The paper introduces ContextWeave, a benchmark for evaluating language agents in long-horizon, stateful workflows.
  • ContextWeave consists of 1,005 executable tasks reconstructed from multi-month workflows of 14 participants.
  • The benchmark measures workspace quality and alignment with participant-specific preferences.
  • It includes diagnostics for relevance, continuity, solvability, and robustness to misleading recall.
  • The study tests six memory components under a fixed model.
  • The strongest configuration raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60.
  • With a fixed memory component, recall improves both outcomes for all five tested base models.
  • Gains vary substantially depending on the base model.
  • Actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries.
  • However, it can also be more susceptible to misleading recall.
  • The findings suggest that memory systems should optimize not only retrieval relevance but also reliable use during execution.

Key Concepts Mentioned

  • ContextWeave
  • language agents
  • long-horizon workflows
  • memory components
  • workspace quality
  • participant-specific preferences
  • diagnostics
  • relevance
  • continuity
  • solvability
  • robustness
  • misleading recall
  • memory systems

Claims Made by the Document

  • ContextWeave is a new benchmark for evaluating language agents in long-horizon, stateful workflows.
  • Actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries.
  • Memory systems should optimize not only retrieval relevance but also reliable use during execution.

Evidence Used

  • The paper describes the creation of ContextWeave, a benchmark for evaluating language agents in long-horizon, stateful workflows.
  • The text mentions that actionable, experience-rich memory improves workflow continuation and reduces redundant exploration compared to compact summaries.
  • The paper concludes by suggesting that memory systems should optimize not only retrieval relevance but also reliable use during execution.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Memory is essential as language agents move from isolated tasks to long-horizon, stateful workflows, yet existing evaluations often reduce it to retrieval or question answering. We introduce ContextWeave, a longitudinal benchmark that evaluates whether recalled experience improves downstream agent performance in realistic office-work streams. ContextWeave reconstructs privacy-preserved, multi-month workflows of 14 participants into 1,005 executable tasks, including 568 core evaluation tasks, with instructions, containerized environments, trajectories, and task-specific rubrics. It measures workspace quality and alignment with participant-specific preferences, complemented by diagnostics of relevance, continuity, solvability, and robustness to misleading recall. Across six memory components under a fixed model, the strongest configuration raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60. With a fixed memory component, recall improves both outcomes for all five tested base models, although gains vary substantially. Our analysis shows that actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries, while it can also be more susceptible to misleading recall. These findings motivate memory systems that optimize not only retrieval relevance but also reliable use during execution.