Enhancing Rubric-based RL via Self-Distillation

Source

Source name: arXiv
Source kind: paper
Authors / organization: Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang
Venue: arXiv: cs.LG
Published: 2026-07-20
Collected: 2026-07-21
URL: http://arxiv.org/abs/2607.18082v1
PDF: https://arxiv.org/pdf/2607.18082v1
DOI: UNKNOWN

Plain-Language Brief

The paper proposes a new method called Criterion-Distilled Policy Optimization (CriPO) to address two limitations in Rubric-based Reinforcement Learning (RL). The first limitation, Limited Exploration, occurs when criteria that no rollout manages to satisfy receive no optimization signal. The second limitation, Suppressed Criteria, happens when criteria are satisfied by some rollouts but their learning signals are lost during optimization. CriPO aims to solve both issues without introducing a training-inference mismatch. It does this by constructing a criterion-injection self-teacher for Limited Exploration and employing a counterfactual self-teacher for Suppressed Criteria.

What the Document Says

Rubric-based RL is used to improve Language Models (LLMs) on open-ended tasks.
A limitation of rubric-based RL is Limited Exploration where criteria that no rollout manages to satisfy receive no optimization signal.
Recent methods address Limited Exploration by incorporating rubric information as external guidance during rollout but introduce a train-inference mismatch.
Another failure mode, Suppressed Criteria, occurs when criteria are satisfied by some rollouts but their learning signals are lost during optimization.
Over 57% of samples exhibit the Suppressed Criteria failure mode throughout training, with an average of 1.8 SC per sample.
CriPO, the proposed method, aims to address both Limited Exploration and Suppressed Criteria without introducing a training-inference mismatch.
For Limited Exploration, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy.
For Suppressed Criteria, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values.
Experiments on medicine and science benchmarks demonstrate that CriPO outperforms rubric-based RL, achieving stronger final performance with approximately $2 imes$ fewer optimization steps.

Key Concepts Mentioned

Rubric-based Reinforcement Learning
Limited Exploration
Suppressed Criteria
Criterion-Distilled Policy Optimization (CriPO)

Claims Made by the Document

Rubric-based RL improves LLMs on open-ended tasks.
Over 57% of samples exhibit the Suppressed Criteria failure mode throughout training.
CriPO consistently outperforms rubric-based RL.

Evidence Used

Rubric-based RL is used to improve Language Models (LLMs) on open-ended tasks.
Over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample.
Experiments on medicine and science benchmarks demonstrate that CriPO outperforms rubric-based RL.

Methods / Approach

Criterion-injection self-teacher
Localized forward-KL loss
Counterfactual self-teacher

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately $2\times$ fewer optimization steps.