Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

Source

Source name: arXiv
Source kind: paper
Authors / organization: Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou
Venue: arXiv: cs.LG
Published: 2026-08-19
Collected: 2026-08-20
URL: http://arxiv.org/abs/2608.19181v1
PDF: https://arxiv.org/pdf/2608.19181v1
DOI: UNKNOWN
Registry group: academic_apis_and_preprints
Evidence role: canonical_academic_index
Language: en
Country / region: UNKNOWN

Plain-Language Brief

Researchers introduce Group-Calibrated On-Policy Distillation (GC-OPD), a method for long-context reasoning tasks that aims to address the mismatch between on-policy distillation and task-specific verifiers. The new approach uses group normalization of verifier rewards and trajectory-level OPD scores, and distributes a signed teacher-verifier disagreement residual across tokens based on their relative OPD advantages. The results show that GC-OPD improves the performance on five long-context benchmarks compared to vanilla OPD.

What the Document Says

On-policy distillation (OPD) is a method where a student model learns from its own responses guided by a stronger teacher. However, in long-context tasks, this approach can lead to locally plausible but incorrect responses due to token-level teacher support not considering evidence across the input or violating global task constraints.
Task-specific verifiers evaluate task completion at the response level and return graded rewards reflecting partial success. There is a mismatch between OPD scores and verifier rewards in longer input ranges, indicating disagreement between the teacher and the verifier.
Group-Calibrated On-Policy Distillation (GC-OPD) separates normalization of verifier rewards and trajectory-level OPD scores within each rollout group. The difference is used as a signed teacher-verifier disagreement residual.
Relative-advantage-based credit assignment (RACA) distributes this residual across tokens according to their relative OPD advantages while preserving the original OPD signal.
The new approach improves the performance on five long-context benchmarks compared to vanilla OPD.

Key Concepts Mentioned

On-policy distillation
long-context reasoning tasks
task-specific verifiers
Group-Calibrated On-Policy Distillation (GC-OPD)
Relative-advantage-based credit assignment (RACA)

Claims Made by the Document

Token-level teacher support can favor locally plausible but incorrect responses in long-context tasks.
Teacher-verifier disagreement occurs in longer input ranges.
Group-Calibrated On-Policy Distillation (GC-OPD) improves the performance on five long-context benchmarks compared to vanilla OPD.

Evidence Used

Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards
Post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively
Vanilla OPD reaches 39.31 and 43.56 under the same setup.

Methods / Approach

Group normalization of verifier rewards and trajectory-level OPD scores
Signed teacher-verifier disagreement residual
Relative-advantage-based credit assignment (RACA)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

It is unknown if the authors have considered all possible limitations of their method.

Open Questions

What are the potential drawbacks or limitations of the proposed Group-Calibrated On-Policy Distillation (GC-OPD) approach?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text does not provide information on potential limitations of the proposed method. It is also unclear if the authors have considered all possible drawbacks.

Raw Abstract / Excerpt

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.