Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

Source

arXiv

arxiv.org/abs/2606.26091v1

Read the source

Source

Plain-Language Brief

The paper discusses a method called On-policy Self-Distillation, which uses a single model as both teacher and student to achieve high accuracy in pass@1. However, it is found that this method can lead to a decrease in rollout diversity, causing the pass@k curves to flatten, and making the generated rollouts less effective at improving accuracy. This is attributed to biases in the design of self-distillation with sampled demonstrations.

What the Document Says

  • On-policy Self-Distillation uses a single model as both teacher and student to achieve high pass@1 accuracy.
  • The teacher conditions its feedback on a correct demonstration in self-distillation.
  • Rollout diversity decreases and pass@k curves flatten with On-policy Self-Distillation.
  • This is due to compounding biases in the design of self-distillation with sampled demonstrations.
  • The teacher scores each student rollout while conditioned on a sampled correct rollout.
  • Self-distillation tilts the base distribution towards certain modes, unlike ideal optimal on-policy reinforcement learning which preserves probability ratios among equally correct rollouts.
  • On controlled graph path-finding tasks and science question-answering benchmarks, self-distilled models perform as well or better than reinforcement learning but have lower functional and semantic diversity.
  • Self-distilled models struggle on out-of-distribution settings that require diverse strategies.

Key Concepts Mentioned

  • On-policy Self-Distillation
  • reinforcement learning
  • rollout diversity
  • compounding biases
  • sampled demonstrations
  • graph path-finding tasks
  • science question-answering benchmarks
  • out-of-distribution settings

Claims Made by the Document

  • On-policy Self-Distillation achieves high pass@1 accuracy.
  • Rollout diversity decreases and pass@k curves flatten with On-policy Self-Distillation due to biases in the design.
  • Self-distilled models perform as well or better than reinforcement learning but have lower functional and semantic diversity.
  • Self-distilled models struggle on out-of-distribution settings that require diverse strategies.

Evidence Used

  • The paper describes the method of On-policy Self-Distillation and its impact on accuracy and rollout diversity.
  • The paper provides theoretical analysis and empirical evidence on the performance of self-distilled models compared to reinforcement learning.
  • The paper compares the performance of self-distilled models and reinforcement learning on controlled tasks and benchmarks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy). We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases. We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct rollout used as context. Unlike the ideal optimal on-policy reinforcement learning (RL), which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps, concentrating mass on already-dominant modes. On a controlled graph path-finding task and science question-answering benchmarks, self-distilled models match or exceed RL on average performance but exhibit substantially lower functional and semantic diversity, failing on out-of-distribution settings that require diverse strategies.