Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note4 min read

H$^2$SD: Hybrid Hindsight Self-Distillation

Source

arXiv

arxiv.org/abs/2607.18955v1

Read the source

Source

Plain-Language Brief

The paper introduces H$^2$SD, a framework for reinforcement learning with verifiable rewards (RLVR) that addresses the limitations of previous methods. H$^2$SD uses a hybrid approach to self-distillation, where it modulates update magnitudes based on a teacher model's probabilities for successful trajectories and conditions the teacher on a reference hint for failed ones. The method aims to improve performance on challenging reasoning tasks while maintaining stability and efficiency.

What the Document Says

  • RLVR has improved large language models' reasoning capabilities for tasks like mathematical reasoning and code generation.
  • Most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment.
  • On-policy distillation (OPD) provides denser supervision but requires an additional teacher and assumes a shared vocabulary.
  • On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy.
  • Directly matching the teacher distribution may cause information leakage and unstable optimization in OPSD.
  • RLSD avoids direct matching but cannot provide an explicit correction direction when the sampled reasoning fails.
  • H$^2$SD is a hybrid hindsight self-distillation framework that uses the teacher differently according to trajectory correctness.
  • For successful trajectories, H$^2$SD modulates update magnitudes based on the teacher's probabilities without changing the direction determined by the reward.
  • For failed trajectories, H$^2$SD conditions the teacher on a reference hint containing key reasoning steps and a verified answer.
  • H$^2$SD outperforms RLVR, OPSD, and RLSD baselines on multiple challenging reasoning benchmarks while maintaining stability and efficiency.

Key Concepts Mentioned

  • Reinforcement learning with verifiable rewards
  • On-policy distillation
  • On-policy self-distillation
  • Hybrid hindsight self-distillation
  • Trajectory correctness

Claims Made by the Document

  • RLVR improves large language models' reasoning capabilities.
  • OPD provides denser supervision but requires an additional teacher and a shared vocabulary.
  • OPSD removes the dependency on an additional teacher.
  • Directly matching the teacher distribution may cause information leakage and unstable optimization in OPSD.
  • RLSD avoids direct matching but cannot provide an explicit correction direction when reasoning fails.
  • H$^2$SD outperforms RLVR, OPSD, and RLSD baselines on challenging reasoning benchmarks.

Evidence Used

  • The text discusses the improvements of RLVR
  • The text explains the issues with OPD
  • The text introduces OPSD
  • The text mentions the challenges with directly matching the teacher distribution in OPSD
  • The text explains the limitations of RLSD
  • The text presents the results showing the performance of H$^2$SD

Methods / Approach

  • H$^2$SD uses hybrid hindsight self-distillation
  • For successful trajectories, H$^2$SD modulates update magnitudes based on the teacher's probabilities
  • For failed trajectories, H$^2$SD conditions the teacher on a reference hint

Limitations Stated by the Authors

  • UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not discuss potential limitations in the real-world application of H$^2$SD
  • The text focuses on challenging reasoning benchmarks, but the performance on other tasks is unknown

Open Questions

  • What are the specific challenges and benefits of H$^2$SD in real-world applications?
  • How does H$^2$SD perform on various other tasks beyond challenging reasoning benchmarks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on improving reinforcement learning with verifiable rewards for mathematical reasoning and code generation. It introduces a new method, H$^2$SD, that addresses the limitations of previous methods by using a hybrid approach to self-distillation.
  • H$^2$SD aims to improve performance on challenging reasoning tasks while maintaining stability and efficiency.

Raw Abstract / Excerpt

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce $\mathrm{H}^{2}\mathrm{SD}$, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H$^2$SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.