Mismatch Matters: On-Policy Distillation Beyond Token Agreement

Source

Source name: arXiv
Source kind: paper
Authors / organization: Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, Difan Zou
Venue: arXiv: cs.AI
Published: 2026-08-10
Collected: 2026-08-11
URL: http://arxiv.org/abs/2608.09836v1
PDF: https://arxiv.org/pdf/2608.09836v1
DOI: UNKNOWN
Registry group: academic_apis_and_preprints
Evidence role: canonical_academic_index
Language: en
Country / region: UNKNOWN

Plain-Language Brief

Researchers present TIDE, a new method to address a failure mode in on-policy distillation (OPD) called degenerate agreement. This issue arises when students exploit repetitive loops to achieve high token agreement with the teacher, but still produce globally flawed responses. The new approach focuses on correcting teacher-student mismatch instead of agreement, categorizing it into two types: student-excess tokens and student-deficit tokens. TIDE uses bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass. The study demonstrates that TIDE outperforms standard OPD and recent token-selection and reward-shaping baselines across mathematical reasoning benchmarks, especially under strong teacher-student mismatch.

What the Document Says

On-policy distillation (OPD) is a core component of modern Language Model post-training pipelines. However, a failure mode called degenerate agreement has been identified, where students exploit repetitive loops to achieve high token agreement with the teacher, despite producing globally flawed responses.
The researchers focus on correcting teacher-student mismatch instead of agreement, and categorize it into two types: student-excess tokens (generated by the student but assigned near-zero probability by the teacher) and student-deficit tokens (preferred by the teacher but rarely sampled by the student).
To tackle these mismatch directions, they propose TIDE (Token-level Independent Deficit-Excess correction), which uses bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass without requiring deficit tokens to be sampled.
The study demonstrates that TIDE outperforms standard OPD and recent token-selection and reward-shaping baselines across mathematical reasoning benchmarks, especially under strong teacher-student mismatch.

Key Concepts Mentioned

On-policy Distillation
Degenerate Agreement
Teacher-Student Mismatch
Token-level Independent Deficit-Excess correction (TIDE)
Bounded Hellinger Shaping
Analytic Teacher Top-$K$ Injection

Claims Made by the Document

Degenerate agreement is a failure mode in on-policy distillation.
TIDE outperforms standard OPD and recent token-selection and reward-shaping baselines.

Evidence Used

The text mentions that TIDE outperforms other methods according to the results of the study.

Methods / Approach

Bounded Hellinger Shaping
Analytic Teacher Top-$K$ Injection

Limitations Stated by the Authors

Not stated in source.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The research may not have considered all possible limitations. The performance of TIDE might degrade under specific conditions or with different types of models.

Open Questions

What are the specific conditions under which TIDE's performance might degrade?
How does TIDE perform with different types of language models?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The research focuses on mathematical reasoning benchmarks, so it is unclear how well TIDE would perform in other domains.

Raw Abstract / Excerpt

On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses. We therefore shift our focus from agreement to teacher-student mismatch, and find that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens. Student-excess tokens are generated by the student but assigned near-zero probability by the teacher; their log-ratio corrections grow unbounded and destabilize the update. Student-deficit tokens, in contrast, are preferred by the teacher but rarely sampled by the student; their absence blocks the transfer of the teacher's reasoning patterns. To tackle these mismatch directions, we propose TIDE (Token-level Independent Deficit-Excess correction), which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass without requiring deficit tokens to be sampled. Across mathematical reasoning benchmarks with multiple Qwen3 teacher-student pairs, TIDE consistently outperforms standard OPD and recent token-selection and reward-shaping baselines. Moreover, the gains of TIDE are more pronounced under strong teacher-student mismatch, where it improves Avg@8 from 6.9% to 20.3%, reduces average response length by a factor of 3.6, and substantially reduces formatting failures. Code is available at https://github.com/yzc-666/TIDE