Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Source facts

The preprint evaluates agents as configurable systems whose task information, reasoning settings, self-verification, time budget, and backbone model all affect outcomes. On four scientific tasks, the abstract reports substantial run-to-run variability, with about 54% of outcome variance coming from repeating the same configuration. It reports that information had the largest effect and that a dedicated verification tool changed behavior more than a prompt asking the agent to verify.

Analysis

The paper supports evaluating the whole agent configuration rather than attributing performance to model identity alone. This has implications for benchmark comparability, procurement, and the design of agent products whose harness choices may be as important as their underlying model.

Uncertainty/Doubt

• This is an arXiv preprint. The variance decomposition and configuration effects were reported on four scientific tasks and were not independently reproduced here.