Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

Source

arXiv

arxiv.org/abs/2607.12640v1

Read the source

Source

Plain-Language Brief

The research investigates the effectiveness of Reinforcement learning with verifiable rewards and Group Relative Policy Optimization (GRPO) on a small language and vision-language model web agent at scales ranging from 4B to 8B. The study finds that GRPO does not significantly improve the performance of the agent on tasks it has already mastered, and in some cases, it even makes the performance worse, particularly on the text track with moderate to high learning rates. The authors explain this failure by identifying two regimes: a degrade regime where middle learning rates negatively impact the agent, and a collapse regime where high learning rates cause the agent to fail. They also find that this coupling between learning rate and performance is specific to the smaller model (4B), but not present in the larger one (8B).

What the Document Says

  • Reinforcement learning with verifiable rewards and GRPO are studied.
  • The study involves a small language and vision-language model web agent at scales of 4B to 8B.
  • GRPO does not improve the success rate of a strong supervised baseline on tasks the agent has largely mastered.
  • High learning rates make it worse on the text track.
  • The null (lack of improvement) holds under various conditions, including paired testing, multiple seeds, observations, and scaling the backbone to 8B.
  • GRPO helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one.
  • Two regimes are identified: degrade regime affecting attention and MLP blocks with middle learning rates, and collapse regime with high learning rates.
  • The coupling between learning rate and performance is specific to the smaller model (4B).

Key Concepts Mentioned

  • Reinforcement learning
  • GRPO
  • Group Relative Policy Optimization
  • Language and vision-language model web agent
  • Learning rate
  • KL weight
  • Seed
  • Initialization
  • Clipping
  • Success rate
  • Text track
  • Set-of-Marks
  • Scaling
  • Embedding change

Claims Made by the Document

  • GRPO does not improve the performance of a small language and vision-language model web agent on tasks it has largely mastered.
  • High learning rates make the performance worse on the text track.

Evidence Used

  • No configuration improves the success rate in 18 runs varying parameters.
  • GRPO helps only when there is headroom to climb.
  • The null holds under various conditions, including paired testing, multiple seeds, observations, and scaling.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent. We ask whether it adds skill to a small language and vision-language model web agent at the 4B to 8B scale, or whether it mostly reshapes behavior the supervised model already has. Across a control grid of 18 runs that varies learning rate, KL weight, seed, initialization, and clipping, no configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered. On the text track, moderate to high learning rates make it credibly worse. The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set-of-Marks screenshot observations, and scaling the backbone to 8B; the credible harm is a text-track finding and is only nominal under Set-of-Marks. To show that the null reflects the setting and not a broken pipeline, we run the identical harness, reward, and recipe on tasks whose reward is reachable by sampling, and there the success rate rises by 22 points with a paired interval that excludes zero. GRPO therefore helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one. We then explain the failure. A middle learning rate degrades the agent and a high one collapses it, and the two regimes form a double dissociation: grafting localizes the degrade regime to the attention and MLP blocks, while the collapse regime cannot be traced to any single group, and the embedding change that dominates the weight movement is causally inert. At 4B, effective rank in the late layers tracks capability in both directions; at 8B the two come apart. This coupling is specific to the smaller model, so we report it as scale-dependent.