Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Weak-to-Strong Generalization via Direct On-Policy Distillation

Source

arXiv

arxiv.org/abs/2607.05394v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
  • Venue: arXiv: cs.LG
  • Published: 2026-07-06
  • Collected: 2026-07-07
  • URL: http://arxiv.org/abs/2607.05394v1
  • PDF: https://arxiv.org/pdf/2607.05394v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers propose a method called Direct On-Policy Distillation (Direct-OPD) for improving stronger language models by leveraging the knowledge gained from reinforcement learning on smaller, weaker models. This method allows for reusing the RL supervision signal of the weak model without training an explicit reward model or running sparse-reward RL on the target model. The study shows that Direct-OPD can boost the performance of a large language model (Qwen3-1.7B) on AIME 2024 task by 14.1% in just 4 hours on 8 A100 GPUs, outperforming step-matched direct RL and enabling the sequential composition of multiple policy shifts.

What the Document Says

  • The paper discusses Reinforcement Learning with Verifiable Rewards (RLVR) as a method to improve language model reasoning.
  • Running RL on smaller models is cheaper but training on every new strong model can be expensive.
  • A weak-to-strong alternative is proposed: running RL on a smaller model, then using what was learned to improve a stronger target model.
  • Directly distilling the post-RL weak teacher isn't sufficient because the final policy combines useful RL gains with limitations of the smaller model.
  • The authors propose Direct On-Policy Distillation (Direct-OPD) which transfers the teacher's RL-induced policy shift instead.
  • Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student.
  • The checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states.
  • Direct-OPD reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model.
  • Empirical results show that Direct-OPD consistently improves stronger target models, especially boosting Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100 GPUs.
  • Direct-OPD outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts.
  • The study concludes that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

Key Concepts Mentioned

  • Reinforcement Learning with Verifiable Rewards (RLVR)
  • Direct On-Policy Distillation (Direct-OPD)
  • Language Model Reasoning
  • Model Scaling
  • Policy Shifts

Claims Made by the Document

  • Directly distilling the post-RL weak teacher is not enough.
  • Direct-OPD transfers the teacher's RL-induced policy shift instead.
  • Direct-OPD reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model.
  • Direct-OPD consistently improves stronger target models.

Evidence Used

  • We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead.
  • Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models.
  • This directly reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model.

Methods / Approach

  • Direct On-Policy Distillation (Direct-OPD)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.