Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

Source

arXiv

arxiv.org/abs/2608.04794v1

Read the source

Source

Plain-Language Brief

The research investigates self-distillation (SD), a method used in reinforcement learning with verifiable rewards, where a teacher conditioned on privileged information supplies supervision to a student. The study finds that while SD works well in easy tasks, it fails in difficult ones. The authors attribute this failure to the teacher's bias towards the reference solution, causing the student's loss function to focus on low-information tokens rather than those that determine the answer, resulting in a flatter and less decisive student.

What the Document Says

  • The study reproduces the reported gains of SDPO in its easy setting but finds no improvement when applied to difficult tasks.
  • The failure of SD is attributed to PI bias, where the teacher's per-token target is pulled towards the reference solution rather than general correctness.
  • The student's loss function, trained to match the teacher's target, becomes nearly blind to whether a rollout is correct, penalizing low-information tokens and hesitation required for reasoning.
  • Across various tasks, reasoning modes, model sizes, forms of PI, and under both SDPO and OPSD recipes, the per-token loss falls while validation accuracy does not improve or degrades.

Key Concepts Mentioned

  • self-distillation
  • privileged information
  • PI bias
  • loss function
  • validation accuracy

Claims Made by the Document

  • SD does not improve in difficult tasks compared to easy ones.
  • PI bias is the cause of SD's failure.
  • The student's loss function focuses on low-information tokens and penalizes hesitation required for reasoning.

Evidence Used

  • Reproduction of SDPO's gains in easy setting, followed by application to difficult tasks with no improvement or degradation in validation accuracy.

Methods / Approach

  • Reproducing SDPO's reported gains
  • Applying identical setup to difficult tasks

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.