Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

Source

arXiv

arxiv.org/abs/2608.04794v1

Read the source

Source

Plain-Language Brief

The study investigates self-distillation (SD), a method used in reinforcement learning for efficient teaching by a 'teacher' using privileged information about the answer. The research finds that while SD reproduces reported gains on easy tasks, it fails to improve validation accuracy on difficult tasks across various domains and reasoning modes. The authors attribute this failure to the teacher's bias towards a specific reference solution, leading the student to focus more on low-information tokens rather than those determining the answer, thus resulting in a less decisive student.

What the Document Says

  • Self-distillation (SD) is a compute-efficient alternative to reinforcement learning with verifiable rewards.
  • The teacher in SD is conditioned on privileged information about the answer.
  • Reported gains of SD come from narrow, low-difficulty settings.
  • The study reproduces SDPO's reported gains in its easy setting but finds that it does not improve accuracy on difficult tasks across various domains and reasoning modes.
  • The failure of SD is attributed to PI bias.
  • PI bias causes the teacher's per-token target to be pulled towards a specific reference solution instead of correctness in general.

Key Concepts Mentioned

  • self-distillation
  • reinforcement learning
  • privileged information
  • teacher
  • student
  • PI bias

Claims Made by the Document

  • SD reproduces reported gains on easy tasks but does not improve accuracy on difficult tasks.
  • PI bias is the cause of SD's failure.

Evidence Used

  • Reproduction and application of identical setup to difficult tasks showing no improvement in validation accuracy.
  • Explanation of the failure through a causal chain from the loss to the model it produces.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.