Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Co-Evolving LLM Evaluators and Policies via DynamicRubric

Source

arXiv

arxiv.org/abs/2607.20083v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Beining Wang, Weihang Su, Hongtao Tian, Hao Kong, Tao Yang, Ting Yao, Qingyi Pan, Yueyue Wu, Qingyao Ai, Min Zhang, Yiqun Liu
  • Venue: arXiv: cs.LG
  • Published: 2026-07-22
  • Collected: 2026-07-23
  • URL: http://arxiv.org/abs/2607.20083v1
  • PDF: https://arxiv.org/pdf/2607.20083v1
  • DOI: UNKNOWN

Plain-Language Brief

The researchers propose DynamicRubric, a method that allows large language models to improve by co-evolving evaluators and policies. This method generates weighted binary rubrics for each candidate set and aggregates the results into response-level scores. The experiments show that DynamicRubric improves evaluator performance, provides stronger policy supervision, and demonstrates gains on verifiable reasoning and coding tasks.

What the Document Says

  • Post-training with evaluator feedback is a major mechanism for improving large language models.
  • As policies improve, the quality of sampled responses becomes close.
  • Close candidates create a bottleneck for policy optimization due to collapsed relative evaluator score gaps yielding weak or misleading policy supervision.
  • The directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them.
  • Relative score gaps are identified as the signals that guide policy updates.
  • DynamicRubric is a framework that generates weighted binary rubrics for each candidate set and aggregates the results into response-level scores.
  • Experiments with 8B backbones show that DynamicRubric improves evaluator performance, provides stronger policy supervision, and demonstrates gains on verifiable reasoning and coding tasks.
  • DynamicRubric-optimized policies also show gains in an AI answering scenario deployed in WeChat Search.
  • The results suggest that evaluators should evolve with the policies they supervise.

Key Concepts Mentioned

  • large language models
  • policy optimization
  • evaluator score gaps
  • DynamicRubric
  • response-level scores
  • 8B backbones
  • verifiable reasoning
  • coding tasks
  • WeChat Search

Claims Made by the Document

  • Post-training with evaluator feedback improves large language models.
  • DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines.
  • DynamicRubric-optimized policies show gains on verifiable reasoning and coding tasks.
  • A DynamicRubric-optimized model is deployed in WeChat Search.

Evidence Used

  • The text describes post-training with evaluator feedback as a major mechanism for improving large language models.
  • The text mentions that experiments show improvements in evaluator performance and policy supervision using DynamicRubric.
  • The text states that DynamicRubric-optimized policies demonstrate gains on verifiable reasoning and coding tasks.
  • The text mentions that a DynamicRubric-optimized model is deployed in WeChat Search.

Methods / Approach

  • Post-training with evaluator feedback
  • DynamicRubric framework

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

It's not stated whether the authors have considered or addressed potential limitations of their proposed method.

Open Questions

  • What are the specific details of the baseline methods used for comparison?
  • How is the evaluation of the performance of the DynamicRubric-optimized models conducted?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text mentions that a reward model and a static rubric generator were used as baselines, but no specific details are given.

Raw Abstract / Excerpt

Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.