Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Source

arXiv

arxiv.org/abs/2606.27288v1

Read the source

Source

Plain-Language Brief

The study examines the use of multiple language models (LLMs) to improve accuracy over single models and finds that their performance is limited by a factor called 'beta'. Beta represents the rate at which all models are incorrect on the same query. The authors demonstrate that a single-factor model can underestimate the 'all-wrong' tail for LLMs, but rephrasing questions in free-response format can increase the beta value, indicating more co-failures among models.

What the Document Says

  • Multi-model LLM systems are used to surpass single-model accuracy.
  • The authors introduce a factor 'beta' that limits the gain of these systems. Beta is the rate at which every model is incorrect on the same query.
  • A traditional diagnostic, average pairwise error correlation (rho), cannot identify beta.
  • The Clopper-Pearson bound on beta gives a certificate for the largest gain achievable by any router, vote, or cascade before training.
  • Across 67 models from 21 providers, the single-factor model underestimates the 'all-wrong' tail. The observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula.
  • Re-asking questions in free-response format reopens the 'all-wrong' tail, with beta 0.127.
  • At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA on checkable tasks, but combining models rarely surpasses the single best model without a strong query-level routing signal.
  • Gains come from models failing on different questions, not from adding more models.

Key Concepts Mentioned

  • multi-model language models
  • routing
  • voting
  • cascades
  • fusion
  • mixture-of-agents
  • beta
  • Clopper-Pearson bound
  • all-wrong tail
  • open-ended mathematics

Claims Made by the Document

  • The gain of multi-model LLM systems is capped by a factor called 'beta'.
  • A single-factor model can underestimate the 'all-wrong' tail for LLMs.
  • Re-asking questions in free-response format reopens the 'all-wrong' tail.

Evidence Used

  • For any policy whose output is one member model answer, accuracy cannot exceed one minus beta.
  • A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router.
  • Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail.
  • Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.