Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Vladislav Beliaev
• Venue: arXiv: cs.LG
• Published: 2026-07-08
• Collected: 2026-07-09
• URL: http://arxiv.org/abs/2607.07690v1
• PDF: https://arxiv.org/pdf/2607.07690v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces Agon, a new reinforcement learning approach that trains two reasoning models by having them compete against each other. This competitive grading system aims to improve the quality of thinking rather than just providing correct answers, as it rewards models for out-solving their rivals. The two models take turns drafting and solving solutions, with one model improving upon the work of the other. According to the results presented in the paper, this approach significantly improves performance on difficult mathematical problems compared to traditional single-model reinforcement learning.

What the Document Says

• Agon is a new competitive reinforcement learning approach.
• It involves two reasoning models that compete against each other during training.
• Each model takes turns drafting and solving solutions, with one model improving upon the work of the other.
• The reward system is based on out-solving the rival model.
• The goal is to improve the quality of thinking rather than just providing correct answers.
• The approach doubles GRPO's pass@1 performance on a difficult mathematical problem (DeepMath with Qwen3).
• It also shows improvements on competitive-programming code and across model families (Qwen3.5, Gemma 4).

Key Concepts Mentioned

• reinforcement learning
• competitive models
• reasoning models
• out-solving
• GRPO
• DeepMath
• Qwen3
• Gemma

Claims Made by the Document

• Agon improves the quality of thinking in reasoning models.
• It provides a more effective reward system compared to traditional reinforcement learning.

Evidence Used

• The paper presents results showing that Agon doubles GRPO's pass@1 performance on DeepMath with Qwen3.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.