Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Source

arXiv

arxiv.org/abs/2607.12790v1

Read the source

Source

Plain-Language Brief

The paper presents a method for creating and improving self-evolving agent systems, which are capable of developing, modifying, and discarding their own skills. The authors claim that in many real-world applications, a reliable evaluation metric doesn't exist. They propose a solution where the metric is evolved through a loop that searches for combinations of small drawback detectors over a full evolutionary lifecycle. This evolved metric agrees with a reference set and is audited against an unseen anchor. The paper also introduces Double Ratchet, a co-evolution of the metric with a skill loop that maintains performance in tasks like code generation, text-to-SQL, and report generation. The authors argue that this failure-expecting architecture should be the default wherever no reliable automatic verifier exists.

What the Document Says

  • The paper discusses self-evolving agent systems that improve by creating, revising, and retiring their own skills. However, a reliable evaluation metric isn't always available in real applications.
  • The authors propose an evolved metric that is created through a loop searching for combinations of small drawback detectors over a full evolutionary lifecycle. This metric agrees with a reference set and is audited against an unseen anchor.
  • Double Ratchet is introduced, which co-evolves the metric with a skill loop, maintaining performance in tasks like code generation, text-to-SQL, and report generation.
  • According to the authors, Double Ratchet retains 88-110% of the held-out lift achieved by the same skill loop driven by ground truth or the best available rubric.

Key Concepts Mentioned

  • self-evolving agent systems
  • evaluation metric
  • evolved metric
  • Double Ratchet
  • code generation
  • text-to-SQL
  • report generation

Claims Made by the Document

  • Metrics can be evolved.
  • Double Ratchet recovers what an accurate metric would have enabled.
  • Safety comes from anchor discipline plus outer audits.

Evidence Used

  • The proposed method of evolving metrics and Double Ratchet are detailed in the paper.

Methods / Approach

  • Evolutionary lifecycle for creating and revising skills
  • Composition of small drawback detectors for creating the evolved metric.
  • Double Ratchet for co-evolving the metric with a skill loop.

Limitations Stated by the Authors

  • The authors acknowledge that removing anchor guards collapses the metric into a vacuous detector.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • Not stated in source
  • It is unclear how the performance of Double Ratchet compares to other methods for co-evolving metrics and skills.

Open Questions

  • How does the proposed method perform compared to other methods for creating self-improving agent systems?
  • What are the long-term implications of using failure-expecting architecture in different scenarios?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on improving the performance of self-evolving agent systems, particularly in situations where a reliable evaluation metric doesn't exist.

Raw Abstract / Excerpt

Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make three claims. First, metrics can be \emph{evolved}: our metric loop searches compositions of small drawback detectors under a full evolutionary lifecycle, trained to agree with a ten-item anchored reference set, regularized by consensus over unlabeled outputs, and audited against a held-out anchor it never reads, yielding a transparent, inspectable metric rather than an opaque judge. Second, since no metric exists to beat, the yardstick is recovering what an accurate metric would have enabled, and \emph{Double Ratchet}, our co-evolution of the metric with a lifecycle-managed skill loop, does so: across code generation (MBPP+), enterprise text-to-SQL (Spider~2.0-Snow), and reference-free report generation, it retains 88--110% of the held-out lift achieved by the same skill loop driven by ground truth or the best available rubric. Third, safety comes from anchor discipline plus outer audits: removing anchor guards collapses the metric into a vacuous detector while removing the lifecycle does not; and when evolved skills gamed the report rubric, an independent judge caught it, one detector repaired it, and a task-aware judge then preferred the evolved outputs over the pre-evolution baseline in 77% of decided pairs. We argue this failure-expecting architecture is the right default wherever no reliable automatic verifier exists.