Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

Source

arXiv

arxiv.org/abs/2607.18006v1

Read the source

Source

Plain-Language Brief

The paper introduces MADA-RL, a method for fine-tuning compact language models with limited budgets. It specializes the models into generator and critic roles and trains them using a debate-aware learning signal. The method uses a counterfactual critic advantage to optimize critics to improve over generator consensus, resulting in more targeted credit assignment. MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model on five mathematical reasoning benchmarks and requires 16 times fewer trainable parameters than fully fine-tuned baselines.

What the Document Says

  • MADA-RL is a post-training framework for compact models
  • The framework specializes the models into generator and critic roles
  • It trains them with a debate-aware learning signal
  • The training uses LoRA adapters to fine-tune only a small subset of parameters
  • The counterfactual critic advantage is used to redefine the critic's advantage as its reward minus the generator ensemble's per-instance accuracy
  • This optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer
  • MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model on five mathematical reasoning benchmarks
  • The accuracy increases by 2.0 points, with a significance level of p < 0.001
  • MADA-RL requires 16 times fewer trainable parameters than fully fine-tuned baselines
  • It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3)

Key Concepts Mentioned

  • MADA-RL
  • compact models
  • generator and critic roles
  • debate-aware learning signal
  • LoRA adapters
  • counterfactual critic advantage
  • DeepSeek-R1-Distill-Qwen-1.5B
  • mathematical reasoning benchmarks

Claims Made by the Document

  • MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model
  • MADA-RL requires fewer trainable parameters than fully fine-tuned baselines

Evidence Used

  • Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from 39.9% to 41.9%

Methods / Approach

  • Post-training framework
  • Specialization into generator and critic roles
  • Debate-aware learning signal
  • LoRA adapters
  • Counterfactual critic advantage

Limitations Stated by the Authors

  • MADA-RL does not surpass the strongest baselines (DeepScaleR, STILL-3)
  • The gap between MADA-RL and the strongest baselines is associated with inference-time cost

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not provide information on the specific inference-time costs or the exact size of the datasets used for training the strongest baselines

Open Questions

  • What are the specific inference-time costs associated with MADA-RL and the strongest baselines?
  • What is the exact size of the datasets used for training the strongest baselines?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper mentions that there is a gap between MADA-RL and the strongest baselines (DeepScaleR, STILL-3), but does not provide specific details on the reasons for this gap
  • The paper mentions an associated inference-time cost, but does not provide specific details on what this cost entails

Raw Abstract / Excerpt

Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 , \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 , %$ to $41.9 , %$ ($+2.0$ points, $p < 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.