Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Xing Zhang, Yanwei Cui, Guanghui Wang, Zhihao Lin, Peiyang He
• Venue: arXiv: cs.AI
• Published: 2026-08-19
• Collected: 2026-08-20
• URL: http://arxiv.org/abs/2608.18744v1
• PDF: https://arxiv.org/pdf/2608.18744v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers developed a method called EvalCEGAR to automatically create an evaluation metric for AI agents in tasks like report generation, where traditional metrics are difficult to develop. The method involves creating a pool of small Python operators that flag issues with candidate answers, and uses counterexample-guided abstraction refinement from program verification to improve the operators. On a test dataset (MBPP+ and HumanEval+), the method was able to create an operator that improved the performance of AI agents on unseen tasks by 15.4%, outperforming hand-written operators in some cases.

What the Document Says

• The research proposes a method called EvalCEGAR for creating evaluation metrics for AI agents.
• EvalCEGAR involves a pool of small Python operators that flag issues with candidate answers.
• Counterexample-guided abstraction refinement from program verification is used to improve the operators.
• The method finds a pair of answers that the operators score identically, one correct and one not, as the authoring request.
• On MBPP+ and HumanEval+, the method created a 55-line operator that improved performance on unseen tasks by 15.4%.
• The operator performed similarly to a hand-written operator on a third of the flags in a benchmark it had never seen.
• Six out of eight runs admitted such an operator, and all six helped with unseen tasks.

Key Concepts Mentioned

• AI agents
• evaluation metric
• counterexample-guided abstraction refinement
• program verification
• operators

Claims Made by the Document

• EvalCEGAR can create an evaluation metric for AI agents.
• The method improves the performance of AI agents on unseen tasks.

Evidence Used

• Creation of a 55-line operator that closed 15.4% of the gap between flagging nothing and a perfect filter on 428 unseen tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Agents improve quickly against a reliable automatic metric and stall without one, and the applications that need them most, report generation among them, are the ones nobody knows how to score. Can the metric write itself? Saying what makes an answer good is hard; pointing at something wrong with one is easier, so the metric we evolve is a pool of small Python operators that each flag a candidate for one named defect, or abstain, and vote. Asking a model for operators directly does not work: 183 candidates realise only 96 distinct behaviours, from one narrow region of an enormous space. EvalCEGAR instead borrows counterexample-guided abstraction refinement from program verification. It reads the pool as an abstraction and searches for a collision, two answers the operators score identically, one correct and one not. That pair, not a prompt, is the authoring request, and when a collision defeats every attempt the loop widens what an operator may read rather than resampling. On MBPP+ and HumanEval+, a sandbox whose hidden unit tests give exact ground truth, the loop writes a 55-line operator that closes 15.4% of the gap between flagging nothing and a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of our best hand-written operator's flags. On the benchmark it never saw it matches that operator's effect exactly on a third of the flags. Six of eight runs admit such an operator and all six help out of sample; our 15 hand-written operators applied together as one filter lose accuracy. An LLM judge on the same information ties that delta on a nearly disjoint set of candidates, and charges a model call per candidate forever where the operator charges none.