A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

Source facts

The study tests whether scoring signals in on-policy self-distillation localize errors and push probability away from erroneous student tokens. It reports that full-distribution KL and token log-ratio signals do neither reliably. A critique-conditioned teacher is then proposed; the abstract reports improved error localization, corrective shifts within error spans, and benefits at test time and in cold-start GRPO. The record identifies the work as an oral at the FLLMPT workshop, NeurIPS 2026.

Analysis

The central distinction is between scoring a rollout and searching for the location and nature of an error. The result suggests that privileged information is not enough for corrective training unless the teacher is explicitly asked to produce an intermediate critique.

Uncertainty/Doubt

• This is an arXiv preprint. The intervention results, training comparisons, and workshop acceptance were not independently reproduced or checked against proceedings here.