Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Source

arXiv

arxiv.org/abs/2607.02502v1

Read the source

Source

Plain-Language Brief

The paper introduces DemoPSD, a novel framework for training large language models (LLMs) that addresses the issues of overfitting, suppression of exploration, cross-domain generalization problems, and privileged information leakage in on-policy self-distillation. It achieves this by steering the student towards a reverse-KL barycenter target, balancing learning from the teacher with preserving the student's own reasoning capacity. The framework was tested on SciKnowEval across four scientific fields and outperformed GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.

What the Document Says

  • On-policy self-distillation (OPSD) is a method for training large language models (LLMs).
  • OPSD's teacher's dense token-level supervision can lead to overfitting, suppress exploration, hurt cross-domain generalization, and introduce privileged information leakage.
  • DemoPSD introduces selective adoption of teacher guidance to resolve these issues.
  • DemoPSD steers the student towards a reverse-KL barycenter target, balancing learning from the teacher with preserving the student's own reasoning capacity.
  • The discrepancy between the teacher and student distributions is used to adaptively control the blending at each token position.
  • DemoPSD achieves leakage attenuation (effective mitigation of privileged information leakage) and exploration preservation (preservation of exploration capacity under dense token-level distillation).
  • Experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy.
  • DemoPSD robustly generalizes to out-of-distribution GPQA benchmarks.

Key Concepts Mentioned

  • On-policy self-distillation
  • large language models
  • teacher's dense token-level supervision
  • privileged information leakage
  • DemoPSD
  • reverse-KL barycenter target
  • leakage attenuation
  • exploration preservation
  • SciKnowEval
  • GPQA benchmarks

Claims Made by the Document

  • OPSD can lead to overfitting, suppress exploration, hurt cross-domain generalization, and introduce privileged information leakage.
  • DemoPSD resolves these issues through selective adoption of teacher guidance.
  • DemoPSD achieves leakage attenuation and exploration preservation.
  • DemoPSD outperforms GRPO and SDPO on SciKnowEval across four scientific fields.
  • DemoPSD maintains higher training entropy.

Evidence Used

  • The text states that recent studies have found certain issues in OPSD, and DemoPSD is introduced to address these issues.

Methods / Approach

  • Steering the student towards a reverse-KL barycenter target
  • Adaptively controlling the blending at each token position based on the discrepancy between the teacher and student distributions

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not discuss potential limitations explicitly. However, as an assistant, I can suggest that future work could investigate the applicability of DemoPSD to other domains or model architectures.

Open Questions

  • What specific mechanisms are used in DemoPSD to achieve leakage attenuation and exploration preservation?
  • How does DemoPSD perform on out-of-distribution GPQA benchmarks compared to other state-of-the-art methods?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on DemoPSD's ability to address certain issues in OPSD and its performance on SciKnowEval across four scientific fields. It does not provide detailed explanations about the mechanisms used by DemoPSD to achieve leakage attenuation and exploration preservation, nor does it compare its performance extensively with other state-of-the-art methods on out-of-distribution GPQA benchmarks.

Raw Abstract / Excerpt

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.