Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Source

arXiv

arxiv.org/abs/2607.02502v1

Read the source

Source

Plain-Language Brief

The paper introduces DemoPSD, a framework designed to address issues in on-policy self-distillation (OPSD) for training large language models. The main problems being addressed are overfitting, suppression of exploration, cross-domain generalization issues, and privileged information leakage. DemoPSD achieves this by steering the student towards a reverse-KL barycenter target, a combination of teacher and student distributions, and adapting the blending at each token position based on the difference between their distributions. The paper claims that DemoPSD achieves leakage attenuation (mitigation of privileged information leakage) and exploration preservation (preservation of exploration capacity under dense token-level distillation). Experiments show that DemoPSD outperforms GRPO and SDPO, maintains higher training entropy, and generalizes well to out-of-distribution GPQA benchmarks.

What the Document Says

  • On-policy self-distillation (OPSD) is a method for training large language models. The teacher in OPSD provides dense token-level supervision, which can lead to overfitting, suppression of exploration, and cross-domain generalization issues, as well as privileged information leakage.
  • DemoPSD is a novel framework that addresses these problems by steering the student towards a reverse-KL barycenter target, a combination of teacher and student distributions. The blending at each token position is adapted based on the difference between their distributions.
  • The paper claims that DemoPSD achieves leakage attenuation (mitigation of privileged information leakage) and exploration preservation (preservation of exploration capacity under dense token-level distillation).
  • Experiments show that DemoPSD outperforms GRPO and SDPO, maintains higher training entropy, and generalizes well to out-of-distribution GPQA benchmarks.

Key Concepts Mentioned

  • On-policy self-distillation
  • large language models
  • teacher-student model
  • privileged information leakage
  • leakage attenuation
  • exploration preservation

Claims Made by the Document

  • DemoPSD achieves leakage attenuation and exploration preservation.
  • DemoPSD outperforms GRPO and SDPO.
  • DemoPSD maintains higher training entropy.

Evidence Used

  • The paper's claims are stated directly in the text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.