Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

Source

arXiv

arxiv.org/abs/2607.07690v1

Read the source

Source

Plain-Language Brief

The paper introduces a new method called Agon for reinforcement learning that allows two competing models to grade each other's reasoning during training. This approach aims to improve the quality of solutions by encouraging models to think better rather than just write more on hard problems.

What the Document Says

  • The paper discusses a new method called Agon for reinforcement learning.
  • The method is based on two competing models that grade each other's reasoning during training.
  • Each model attempts to solve the same problem, with one drafting a solution and the other reading it while solving.
  • Each model is rewarded for out-solving the other.
  • The purpose of this competition is to encourage models to reason better on hard problems.
  • During training, both models face progressively stronger rivals.
  • The method doubles the pass@1 rate on the DeepMath dataset compared to GRPO, and offers a significant improvement over an untrained Mixture-of-Agents approach.
  • The ordering of results replicates across different model families and competitive programming code.

Key Concepts Mentioned

  • reinforcement learning
  • competitive models
  • implicit rival grading
  • reasoning improvement

Claims Made by the Document

  • Agon improves the quality of solutions on hard problems.
  • Agon encourages models to think better instead of writing more.
  • The two-stage cascade deployment of Agon doubles GRPO's pass@1 rate.

Evidence Used

  • The results reported in the paper

Methods / Approach

  • Competitive reinforcement learning with implicit rival grading

Limitations Stated by the Authors

  • For now, the models communicate in text and do not reason together in latent space.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The method requires both models to be comparably strong and behaviorally different.
  • The performance of Agon may vary depending on the specific model families and tasks being used.

Open Questions

  • What improvements can be achieved when allowing the models to reason together in latent space?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper does not provide detailed information about how the reward system works.
  • It is unclear how the behavioral differences between models are determined or ensured.

Raw Abstract / Excerpt

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.