Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Source

arXiv

arxiv.org/abs/2606.27288v1

Read the source

Source

Plain-Language Brief

The study investigates the use of multi-model language models to improve accuracy over single-models. It is found that the improvement is limited by a factor called 'beta', which represents the rate at which all models are incorrect on the same query. The research also highlights that traditional diagnostic methods cannot identify beta, and introduces a Clopper-Pearson bound as a finite-sample certificate to determine the largest gain possible for any routing or voting system before training a router. The study examines 67 language models from various providers and finds that on open-ended mathematics tasks, beta is significantly higher than expected under a full 67-model Gaussian copula.

What the Document Says

  • Multi-model LLM systems are used to surpass single-model accuracy.
  • The improvement is limited by a factor called 'beta', which is the rate at which all models are incorrect on the same query.
  • Traditional diagnostic methods cannot identify beta.
  • A Clopper-Pearson bound is introduced as a finite-sample certificate to determine the largest gain possible for any routing or voting system before training a router.
  • Across 67 models, observed beta on open-ended mathematics tasks is higher than expected under a full 67-model Gaussian copula.
  • On free-response questions, co-failure in answer format is located rather than subject.

Key Concepts Mentioned

  • multi-model language models
  • routing
  • voting
  • cascade
  • fusion
  • mixture-of-agents
  • beta
  • Clopper-Pearson bound

Claims Made by the Document

  • Multi-model systems can beat single-model accuracy.
  • The gain is limited by a factor called beta.
  • Traditional diagnostic methods cannot identify beta.
  • A Clopper-Pearson bound can determine the largest gain possible before training a router.
  • Observed beta on open-ended mathematics tasks is higher than expected under a full 67-model Gaussian copula.
  • Co-failure in answer format is located rather than subject on free-response questions.

Evidence Used

  • The study presents the use of multi-model systems to improve accuracy over single-models.
  • The paper defines and uses 'beta' as a factor limiting the improvement.
  • The authors state that traditional diagnostic methods cannot identify beta.
  • They introduce the Clopper-Pearson bound as a tool to determine the largest gain possible before training a router.
  • Results show that observed beta on open-ended mathematics tasks is higher than expected under a full 67-model Gaussian copula.
  • The study discusses findings from free-response questions and locates co-failure in answer format.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.