Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Source

arXiv

arxiv.org/abs/2606.27288v1

Read the source

Source

Plain-Language Brief

The text discusses research about using multiple language models (LLMs) to improve accuracy over single models. The study reveals that the gain from these systems is limited by a factor called 'beta', which represents the rate at which all models are wrong on the same query. The text also mentions that a tetrachoric-calibrated single-factor model underestimates the all-wrong tail in open-ended mathematics tasks, with beta being 0.052 versus 0.023 under the full 67-model Gaussian copula.

What the Document Says

  • Multi-model LLM systems are used to beat single-model accuracy.
  • Accuracy is capped by a factor 'beta', which is the rate at which every model is wrong on the same query for policies whose output is one member model answer.
  • The average pairwise error correlation rho cannot identify beta.
  • Clopper-Pearson bound can provide a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router.
  • Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model underprices the all-wrong tail in open-ended mathematics tasks, with observed beta being 0.052 versus 0.023 under the full 67-model Gaussian copula.
  • The effect recurs on execution-graded code, where beta is 0.079.

Key Concepts Mentioned

  • multi-model language models
  • routing
  • voting
  • cascades
  • fusion
  • mixture-of-agents
  • beta
  • Clopper-Pearson bound
  • tetrachoric-calibrated single-factor model
  • Gaussian copula
  • open-ended mathematics
  • execution-graded code
  • GPQA-Diamond questions
  • query-level routing signal

Claims Made by the Document

  • The gain from multi-model LLM systems is capped by a factor 'beta'.
  • A tetrachoric-calibrated single-factor model underestimates the all-wrong tail in open-ended mathematics tasks.
  • Combining models rarely beats the single best model without a strong query-level routing signal.

Evidence Used

  • For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query.
  • Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.