Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Source

arXiv

arxiv.org/abs/2606.27288v1

Read the source

Source

Plain-Language Brief

The study discusses multi-model language models and how their accuracy is limited by a factor called 'beta', which is the rate at which all models are wrong on the same query. The authors find that across 67 models, a single-factor model underprices the all-wrong tail, suggesting that co-failure occurs in answer format rather than subject.

What the Document Says

  • Multi-model language models are used to improve accuracy over single models.
  • The authors introduce a factor 'beta' which is the rate at which all models are wrong on the same query, and argue that the gain from these systems is capped by this factor.
  • Pairwise error correlation rho cannot identify beta, but a Clopper-Pearson bound can provide a finite-sample certificate on the largest gain any router could deliver before training.
  • Across 67 models from 21 providers, the observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, indicating underpricing of the all-wrong tail.
  • The effect recurs on execution-graded code with beta 0.079.
  • Re-asking the same questions in free-response format reopens the tail, with beta 0.127.
  • At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA on checkable tasks, but combining models rarely beats the single best model without a strong query-level routing signal.

Key Concepts Mentioned

  • multi-model language models
  • routing
  • voting
  • cascades
  • fusion
  • mixture-of-agents
  • beta
  • Clopper-Pearson bound
  • Gaussian copula
  • tetrachoric calibration
  • GPQA-Diamond questions
  • query-level routing signal

Claims Made by the Document

  • The gain from multi-model language models is capped by a factor 'beta'.
  • Pairwise error correlation rho cannot identify beta.
  • Across 67 models, the observed beta is underpriced by a single-factor model.
  • Co-failure occurs in answer format rather than subject.
  • Low-rho heterogeneous ensembles beat high-rho Self-MoA on checkable tasks.

Evidence Used

  • For any policy whose output is one member model answer, accuracy cannot exceed one minus beta.
  • A Clopper-Pearson bound gives a finite-sample certificate on the largest gain any router could deliver before training.
  • Across 67 models from 21 providers, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula.
  • Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail with beta 0.127.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.