DOPD: Dual On-policy Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang, Shuai Dong, Kaiwen Tuo, Xiangyu Zeng, Kaituo Feng, Qunzhong Wang, Yang Shi, Xiaobin Hu, Xiangyu Yue, Jiaqi Wang, Shuicheng Yan
• Venue: arXiv: cs.AI
• Published: 2026-06-29
• Collected: 2026-06-30
• URL: http://arxiv.org/abs/2606.30626v1
• PDF: https://arxiv.org/pdf/2606.30626v1
• DOI: UNKNOWN

Plain-Language Brief

The paper proposes a new method called Dual On-policy Distillation (DOPD) for transferring knowledge from a teacher model to a student model in on-policy distillation. DOPD aims to address the issue of 'privilege illusion' by dynamically routing token-level supervision between privileged teacher and student policies based on their advantage gap and relative probabilities. This method improves performance consistently over Vanilla OPD and other counterparts in large language model (LLM) and vision-language model (VLM) settings, and also demonstrates superiority in terms of stability, robustness, continual learning, and out-of-distribution tasks.

What the Document Says

• DOPD is a new method for on-policy distillation.
• It aims to alleviate the 'privilege illusion' issue by dynamically routing token-level supervision between privileged teacher and student policies based on their advantage gap and relative probabilities.
• Each token receives supervision of different strength, objective, and strategy from either teacher or student itself.
• Experiments show that DOPD outperforms Vanilla OPD and other counterparts in LLM and VLM settings.
• It also demonstrates superiority in stability, robustness, continual learning, and out-of-distribution tasks.

Key Concepts Mentioned

• on-policy distillation
• DOPD
• privilege illusion
• token-level supervision
• advantage gap
• relative probabilities

Claims Made by the Document

• DOPD outperforms Vanilla OPD and other counterparts.
• DOPD demonstrates superiority in stability, robustness, continual learning, and out-of-distribution tasks.

Evidence Used

• Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.