Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Source

arXiv

arxiv.org/abs/2607.02502v1

Read the source

Source

Plain-Language Brief

DemoPSD is a new framework for training large language models that addresses overfitting, privileged information leakage, and exploration suppression issues in on-policy self-distillation (OPSD). It does this by steering the student towards a reverse-KL barycenter target, which balances learning from the teacher with preserving the student's own reasoning capacity. The framework was tested on SciKnowEval across four scientific fields and showed better performance than GRPO and SDPO, while maintaining higher training entropy and generalizing well to out-of-distribution GPQA benchmarks.

What the Document Says

  • DemoPSD is a novel framework for large language models.
  • It addresses problems in on-policy self-distillation (OPSD) such as overfitting, privileged information leakage, and exploration suppression.
  • The framework steers the student towards a reverse-KL barycenter target to balance learning from the teacher with preserving the student's own reasoning capacity.
  • The discrepancy between their distributions is used to adaptively control the blending at each token position.
  • DemoPSD was tested on SciKnowEval across four scientific fields and outperformed GRPO and SDPO.
  • It maintains higher training entropy and generalizes well to out-of-distribution GPQA benchmarks.

Key Concepts Mentioned

  • DemoPSD
  • On-policy self-distillation (OPSD)
  • large language models (LLMs)
  • teacher distribution
  • student distribution
  • reverse-KL barycenter target
  • privileged information leakage
  • exploration capacity
  • SciKnowEval
  • GPQA benchmarks

Claims Made by the Document

  • DemoPSD resolves problems in OPSD through selective adoption of teacher guidance.
  • DemoPSD achieves leakage attenuation and exploration preservation.
  • DemoPSD outperforms GRPO and SDPO on SciKnowEval.
  • DemoPSD maintains higher training entropy.

Evidence Used

  • The text explains that DemoPSD steers the student towards a reverse-KL barycenter target, which is intended to resolve problems in OPSD.
  • The text mentions that DemoPSD achieves leakage attenuation and exploration preservation.
  • The text states that DemoPSD outperforms GRPO and SDPO on SciKnowEval.
  • The text implies that DemoPSD maintains higher training entropy.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.