Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation

Source

arXiv

arxiv.org/abs/2608.04788v1

Read the source

Source

Plain-Language Brief

Researchers propose Observation-Calibrated Self-Distillation (OCSD) to address a confounding issue in On-Policy Self-Distillation (OPSD), a method used to train large language model agents. OCSD aims to attribute support specifically to the privileged information contained in the replay view, rather than score shifts induced by the replay scaffold. It does this by contrasting two structurally matched replay views, Full and Observation-Ablated, and using the resulting observation residual to modulate token-level updates at high-uncertainty steps. The method outperforms strong baselines in various tasks and aligns better with local environment feedback.

What the Document Says

  • Large language model agents are trained through reinforcement learning with sparse trajectory-level rewards.
  • On-Policy Self-Distillation (OPSD) addresses the lack of guidance on individual token updates by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision.
  • A confounding issue arises in OPSD as the support may reflect both privileged information and score shifts induced by the replay scaffold.
  • The confounding is especially pronounced when future environment observations serve as privileged information due to the extended scaffold reconstruction.
  • Observation-Calibrated Self-Distillation (OCSD) is proposed to resolve this confounding by contrasting two structurally matched replay views, Full and Observation-Ablated, to derive an observation residual.
  • The residual is used to modulate token-level GRPO updates at high-uncertainty steps while preserving the trajectory-level update direction.
  • Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD outperforms strong baselines.
  • Diagnostic analyses confirm that the calibrated residual aligns better with local environment feedback.

Key Concepts Mentioned

  • Large language model agents
  • Reinforcement learning
  • On-Policy Self-Distillation (OPSD)
  • Observation-Calibrated Self-Distillation (OCSD)
  • ALFWorld
  • WebShop
  • Search-QA
  • Qwen3 model scales
  • GRPO updates

Claims Made by the Document

  • OCSD outperforms strong baselines.
  • The calibrated residual aligns better with local environment feedback.

Evidence Used

  • Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD outperforms strong baselines.
  • Diagnostic analyses confirm that the calibrated residual aligns better with local environment feedback.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.