Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note4 min read

TREK: Distill to Explore, Reinforce to Refine

Source

arXiv

arxiv.org/abs/2607.05339v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Yuanda Xu, Zhengze Zhou, Kayhan Behdin, Jelena Markovic-Voronov, Hejian Sang, Xiaomin Li, Wenhui Zhu, Xinchen Du, Aida Rahmattalabi, Ran He, Sen Na, Zhipeng Wang, Alborz Geramifard
  • Venue: arXiv: cs.LG
  • Published: 2026-07-06
  • Collected: 2026-07-07
  • URL: http://arxiv.org/abs/2607.05339v1
  • PDF: https://arxiv.org/pdf/2607.05339v1
  • DOI: UNKNOWN

Plain-Language Brief

The paper proposes a method called TREK (Teacher-Routed Exploration via Forward KL) to improve Group Relative Policy Optimization (GRPO). TREK uses distillation for exploration support expansion and can work with different types of teachers. It identifies hard prompts where the student's pass rate is low, generates candidate solutions, pulls these verified modes into the student's support, and then refines them using standard GRPO. The method shows significant improvements on mathematical reasoning tasks like AIME 2024 and AIME 2025, as well as agentic tasks in ALFWorld and ScienceWorld.

What the Document Says

  • The paper introduces TREK (Teacher-Routed Exploration via Forward KL).
  • TREK is used to support exploration in Group Relative Policy Optimization (GRPO).
  • TREK uses distillation for expanding the student's support.
  • TREK can work with different types of teachers: external black-box, white-box, or the same model given additional context.
  • TREK identifies hard prompts where the unaided student has a low pass rate.
  • TREK generates verified candidate solutions for these hard prompts from a proposal source.
  • The top-$r$ proposals are ranked by the current student's likelihood.
  • A short forward-KL phase is applied to pull these verified modes into the student's support.
  • After this, TREK returns to standard on-policy GRPO refinement.
  • TREK shows significant improvements on mathematical reasoning tasks like AIME 2024 and AIME 2025 with DeepSeek-V4 proposals.
  • For Qwen3-8B, TREK improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16).
  • The self-context variant of TREK reaches 38.5 and 49.6 without an external teacher on the same tasks.
  • On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7.

Key Concepts Mentioned

  • Group Relative Policy Optimization (GRPO)
  • Teacher-Routed Exploration via Forward KL (TREK)
  • distillation
  • exploration support expansion
  • black-box teacher
  • white-box teacher
  • proposal source
  • forward-KL phase
  • mathematical reasoning
  • AIME 2024
  • AIME 2025
  • Qwen3
  • ALFWorld
  • ScienceWorld

Claims Made by the Document

  • TREK improves GRPO on hard prompts whose correct solution modes lie outside the student's on-policy support.
  • TREK can identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable.

Evidence Used

  • On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025.
  • On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7.

Methods / Approach

  • Identifying prompts where the unaided student has low pass rate.
  • Querying a proposal source to produce verified candidate solutions.
  • Ranking proposals by current student likelihood.
  • Applying a short forward-KL phase.
  • Returning to standard on-policy GRPO refinement.

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not provide information about the potential limitations of TREK in terms of computational complexity or scalability.

Open Questions

  • What is the computational complexity and scalability of TREK?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It would be useful to know if there are any limitations in terms of computational complexity or scalability when using TREK.

Raw Abstract / Excerpt

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-$r$ proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.