Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

Source

arXiv

arxiv.org/abs/2607.18006v1

Read the source

Source

Plain-Language Brief

Researchers introduce MADA-RL, a framework that improves reasoning performance in compact models (up to 4 billion parameters) while reducing training cost. This is achieved by specializing the models into generator and critic roles, using a debate-aware learning signal with a dynamic, role-conditioned baseline for more targeted credit assignment. The framework raises accuracy of one model on five mathematical reasoning benchmarks, uses fewer trainable parameters than fully fine-tuned baselines, but still falls short of the strongest baselines which are trained on larger datasets.

What the Document Says

  • MADA-RL is a post-training framework for compact models.
  • The framework specializes models into generator and critic roles.
  • Training is done with a debate-aware learning signal.
  • LoRA adapters are used to fine-tune only a small subset of parameters.
  • A counterfactual critic advantage is introduced, which redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy.
  • At deployment, specialized agents are composed in a lightweight multi-round protocol.
  • Across five mathematical reasoning benchmarks, MADA-RL improves the accuracy of DeepSeek-R1-Distill-Qwen-1.5B model from 39.9% to 41.9%, using fewer trainable parameters than fully fine-tuned baselines.
  • MADA-RL falls short of the strongest baselines (DeepScaleR, STILL-3), which are trained on larger datasets.
  • A controlled study indicates that MADA-RL's gains come from the counterfactual advantage, as it produces the highest critic improvement rate.

Key Concepts Mentioned

  • MADA-RL
  • compact models
  • generator and critic roles
  • debate-aware learning signal
  • LoRA adapters
  • counterfactual critic advantage
  • mathematical reasoning benchmarks
  • DeepSeek-R1-Distill-Qwen-1.5B
  • DeepScaleR
  • STILL-3

Claims Made by the Document

  • MADA-RL improves accuracy of DeepSeek-R1-Distill-Qwen-1.5B model on mathematical reasoning benchmarks.
  • MADA-RL uses fewer trainable parameters than fully fine-tuned baselines.
  • MADA-RL falls short of the strongest baselines (DeepScaleR, STILL-3).

Evidence Used

  • Across five mathematical reasoning benchmarks, MADA-RL raises accuracy...
  • It uses $16$ times fewer trainable parameters than fully fine-tuned baselines...
  • But still does not surpass the strongest baselines (DeepScaleR, STILL-3)...

Methods / Approach

  • Post-training framework for compact models
  • Specialization into generator and critic roles
  • Debate-aware learning signal
  • LoRA adapters
  • Counterfactual critic advantage

Limitations Stated by the Authors

  • MADA-RL falls short of the strongest baselines (DeepScaleR, STILL-3), which are trained on larger datasets.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study does not provide information on how the size of the dataset affects the performance of MADA-RL.

Open Questions

  • What is the impact of MADA-RL on other types of tasks beyond mathematical reasoning?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The research focuses on improving reasoning performance in compact models, but does not discuss how it applies to larger models.

Raw Abstract / Excerpt

Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 , \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 , %$ to $41.9 , %$ ($+2.0$ points, $p < 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.