Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle

Source

arXiv

arxiv.org/abs/2607.13618v1

Read the source

Source

Plain-Language Brief

Researchers introduce a new benchmark called STOCKTAKE for evaluating LLM agents in supply-chain replenishment tasks over 26 weeks. The benchmark separates state estimation and control, measures the agent's ability to detect hidden failures, and compares their skill scores and response rates.

What the Document Says

  • The research focuses on LLM agents in multi-week decision tasks where the final cost cannot explain an agent's failure.
  • STOCKTAKE is a supply-chain replenishment benchmark designed as a factored partially observable Markov decision process with six hidden factor processes.
  • A fair reference policy, computable through an exact Bayes filter per factor, drives rollout policy on the identical observation stream that the agent receives.
  • The score for each run ranges from a symptom-blind base-stock floor (0) to the oracle (1), yielding a skill score.
  • Grading each week's written rationale provides a stated-belief detection lag and a knowing-doing rate.
  • Four models, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5, were tested on STOCKTAKE with curated stress profiles.
  • The models detected 84-88% of hidden failures, typically within a week of onset.
  • Skill scores ranged from 0.62 to -0.23, with two models ending below the symptom-blind floor and naming factors slightly faster than the two that beat it.
  • When stress persists, 34-43% of correctly diagnosed stress weeks still end in stockout for every model.
  • The failure has two faces: under-response and responses whose cost exceeds what they protect.

Key Concepts Mentioned

  • LLM agents
  • STOCKTAKE benchmark
  • supply-chain replenishment
  • factored partially observable Markov decision process
  • hidden factor processes
  • skill score
  • stated-belief detection lag
  • knowing-doing rate

Claims Made by the Document

  • Existing evaluations cannot separate misreading the world and failing to act.
  • STOCKTAKE measures state estimation and control separately.
  • The four tested models can detect 84-88% of hidden failures.

Evidence Used

  • Reference policies either read privileged information the agent never sees, or are missing altogether.
  • Scoring each run between a symptom-blind base-stock floor (0) and this oracle (1) yields a skill score.
  • Grading each week's written rationale provides a stated-belief detection lag and a knowing-doing rate.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed. On such tasks the final cost cannot say why an agent failed: it may have misread the world, or read it correctly and still failed to act (the knowing-doing gap). Existing evaluations cannot separate these two failures; their reference policies either read privileged information the agent never sees, or are missing altogether. We introduce STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored partially observable Markov decision process with six hidden factor processes, designed so that a fair reference policy is computable: an exact Bayes filter per factor drives a rollout policy on the identical observation stream the agent receives. Scoring each run between a symptom-blind base-stock floor (0) and this oracle (1) yields a skill score, and grading each week's written rationale yields a stated-belief detection lag and a knowing-doing rate, so state estimation and control are measured separately. On fifty seeds with curated stress profiles, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5 detect 84-88% of hidden failures, typically within a week of onset, yet span skill scores from 0.62 to -0.23: two of the four end below the symptom-blind floor while naming factors slightly faster than the two that beat it. The failure has two faces. Where stress persists, 34-43% of correctly diagnosed stress weeks still end in stockout for every model, a rate that partly reflects the severity of the weeks models notice. That rate also runs opposite to skill: the two models under the floor stock out least on diagnosed weeks, so under-response is only one face of the gap, and their traces point to the other, responses whose cost exceeds what they protect. STOCKTAKE measures both directions of that failure.