Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

Source

arXiv

arxiv.org/abs/2607.12640v1

Read the source

Source

Plain-Language Brief

The study investigates the impact of Reinforcement learning with verifiable rewards and Group Relative Policy Optimization (GRPO) on a small language and vision-language model web agent. The research found that GRPO does not significantly improve the success rate of the agent on tasks it has already mastered, and in some cases, it makes the text track worse. The study attributes this failure to middle and high learning rates degrading the agent and a high learning rate causing collapse. This effect is specific to the smaller model at 4B but becomes scale-dependent at 8B.

What the Document Says

  • The study investigates the impact of Reinforcement learning with verifiable rewards, specifically Group Relative Policy Optimization (GRPO), on a small language and vision-language model web agent.
  • The null hypothesis holds that GRPO does not significantly improve the success rate of a strong supervised baseline on tasks the agent has largely mastered.
  • On the text track, moderate to high learning rates make it credibly worse.
  • The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set-of-Marks screenshot observations, and scaling the backbone to 8B.
  • GRPO helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one.
  • A middle learning rate degrades the agent and a high one collapses it.
  • The effect of this failure is specific to the smaller model at 4B but becomes scale-dependent at 8B.

Key Concepts Mentioned

  • Reinforcement learning
  • verifiable rewards
  • Group Relative Policy Optimization (GRPO)
  • language and vision-language model web agent
  • learning rate
  • KL weight
  • seed
  • initialization
  • clipping

Claims Made by the Document

  • GRPO does not significantly improve the success rate of a strong supervised baseline on tasks the agent has largely mastered.
  • Moderate to high learning rates make it credibly worse on the text track.

Evidence Used

  • No configuration improves the success rate of a strong supervised baseline on tasks the agent has largely mastered across a control grid of 18 runs.
  • On the text track, moderate to high learning rates make it credibly worse.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent. We ask whether it adds skill to a small language and vision-language model web agent at the 4B to 8B scale, or whether it mostly reshapes behavior the supervised model already has. Across a control grid of 18 runs that varies learning rate, KL weight, seed, initialization, and clipping, no configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered. On the text track, moderate to high learning rates make it credibly worse. The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set-of-Marks screenshot observations, and scaling the backbone to 8B; the credible harm is a text-track finding and is only nominal under Set-of-Marks. To show that the null reflects the setting and not a broken pipeline, we run the identical harness, reward, and recipe on tasks whose reward is reachable by sampling, and there the success rate rises by 22 points with a paired interval that excludes zero. GRPO therefore helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one. We then explain the failure. A middle learning rate degrades the agent and a high one collapses it, and the two regimes form a double dissociation: grafting localizes the degrade regime to the attention and MLP blocks, while the collapse regime cannot be traced to any single group, and the embedding change that dominates the weight movement is causally inert. At 4B, effective rank in the late layers tracks capability in both directions; at 8B the two come apart. This coupling is specific to the smaller model, so we report it as scale-dependent.