Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Source

arXiv

arxiv.org/abs/2607.02502v1

Read the source

Source

Plain-Language Brief

The paper introduces DemoPSD, a framework for training large language models that addresses the issues of overfitting, suppressed exploration, and privileged information leakage in on-policy self-distillation (OPSD). DemoPSD steers the student towards a reverse-KL barycenter target to balance learning from the teacher with preserving the student's own reasoning capacity. The approach has been shown to mitigate privileged information leakage and preserve exploration capacity, outperforming other methods on various scientific evaluation tasks.

What the Document Says

  • DemoPSD is a novel framework for training large language models (LLMs).
  • It resolves issues of overfitting, suppressed exploration, and privileged information leakage in on-policy self-distillation (OPSD).
  • It steers the student towards a reverse-KL barycenter target.
  • The discrepancy between their distributions is used to adaptively control the blending at each token position.
  • DemoPSD has been shown to mitigate privileged information leakage and preserve exploration capacity.
  • It outperforms GRPO and SDPO on SciKnowEval across four scientific fields.
  • It maintains higher training entropy and robustly generalizes to out-of-distribution GPQA benchmarks.

Key Concepts Mentioned

  • DemoPSD
  • On-policy self-distillation (OPSD)
  • large language models (LLMs)
  • teacher's dense token-level supervision
  • privileged information leakage
  • reverse-KL barycenter target
  • exploration capacity
  • SciKnowEval
  • GPQA benchmarks

Claims Made by the Document

  • DemoPSD resolves issues of overfitting, suppressed exploration, and privileged information leakage in OPSD.
  • DemoPSD achieves leakage attenuation and exploration preservation.
  • DemoPSD outperforms GRPO and SDPO on SciKnowEval.
  • DemoPSD maintains higher training entropy and robustly generalizes to out-of-distribution GPQA benchmarks.

Evidence Used

  • The text states that DemoPSD resolves issues of overfitting, suppressed exploration, and privileged information leakage.
  • The text mentions the achievement of leakage attenuation and exploration preservation by DemoPSD.
  • The text states that DemoPSD outperforms GRPO and SDPO on SciKnowEval.
  • The text states that DemoPSD maintains higher training entropy and robustly generalizes to out-of-distribution GPQA benchmarks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.