Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Rodrigo Guedes de Souza, Alison R. Panisson
• Venue: arXiv preprint
• Published: 2026-08-12
• Collected: 2026-08-14
• URL: https://arxiv.org/abs/2608.12150
• PDF: https://arxiv.org/pdf/2608.12150
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

An evaluation study examining how token budgets affect model accuracy and comparative rankings.

What the Document Says

The paper varies the maximum generation budget across seven levels from 64 to 4,096 tokens and evaluates four models on three reasoning benchmarks. The abstract reports non-monotone item-level behavior, model-ranking reversals across budgets, model complementarity in an oracle analysis, and a budget-aware router that recovers part of the oracle gap while showing domain-transfer limitations.

Key Concepts Mentioned

• LLM evaluation
• inference budget
• model routing
• benchmark methodology

Claims Made by the Document

• The abstract reports non-monotone accuracy on 3–19% of items.
• Model rankings reverse across evaluated budgets on all three benchmarks.
• A budget-aware router captures part of the reported oracle gap but budget features can hurt transfer.

Evidence Used

• The source reports 56,476 inferences across four models, three reasoning benchmarks, and seven token budgets.

Methods / Approach

The study evaluates four models across controlled generation budgets and benchmarks, followed by oracle and routing analyses.

Limitations Stated by the Authors

The abstract does not establish how the findings generalize to other model families, tasks, sampling settings, or cost functions.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The note is based on the arXiv abstract and search record, not the full paper.

Open Questions

• Which benchmark and model characteristics drive ranking reversals?
• How should evaluation reports expose budget conditions for reproducibility?

Notes for Future Q&A

May include interpretation. Labeled where it does.

Use this note for questions about budget-conditioned evaluation and inference routing.

Raw Abstract / Excerpt

The paper varies the maximum generation budget across seven levels from 64 to 4,096 tokens and evaluates four models on three reasoning benchmarks. The abstract reports non-monotone item-level behavior, model-ranking reversals across budgets, model complementarity in an oracle analysis, and a budget-aware router that recovers part of the oracle gap while showing domain-transfer limitations.