Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting

Source

arXiv

arxiv.org/abs/2607.18084v1

Read the source

Source

Plain-Language Brief

The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents in predicting football matches. The benchmark was used for the 2026 FIFA World Cup, but can be applied to future leagues and cups as well. Models are given either a common evidence package or allowed to search for information themselves before making predictions about various aspects of the match. The paper compares the performance of different systems on result accuracy, exact-score accuracy, and scoreline score, among other prediction tasks. The best system showed small gains in result and exact-score accuracy compared to betting-market and human-fan baselines, but a clearer gain in Scoreline.

What the Document Says

  • The WorldCupArena is a dynamic benchmark for evaluating language models and deep-research agents on football match predictions.
  • It was first used for the 2026 FIFA World Cup, with plans to apply it to future leagues and cups as well.
  • Before each match, models receive either a common evidence package or are allowed to search for information themselves.
  • The models then make predictions about various aspects of the match such as result, score, likely players, events, match statistics, and the outcome of the competition.
  • After the match, these predictions are compared with the recorded results.
  • Result accuracy, exact-score accuracy, and a scoreline score are reported, along with scores for other prediction tasks.
  • Across 104 matches and 13 systems, models with similar result accuracy show clearer differences on detailed predictions.
  • The paper compares the performance of these systems against betting-market and human-fan baselines.
  • The best system shows small gains in result and exact-score accuracy compared to baselines, but a clearer gain in Scoreline.
  • New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known.
  • Code, prompts, predictions, and evaluation scripts are open-sourced at https://github.com/wzk1015/WorldCupArena.

Key Concepts Mentioned

  • WorldCupArena
  • football match predictions
  • language models
  • deep-research agents
  • 2026 FIFA World Cup
  • result accuracy
  • exact-score accuracy
  • scoreline score

Claims Made by the Document

  • The WorldCupArena can evaluate language models and deep-research agents on football match predictions.
  • The best system shows small gains in result and exact-score accuracy compared to baselines, but a clearer gain in Scoreline.

Evidence Used

  • Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available.
  • We present WorldCupArena, a dynamic benchmark for language models and deep-research agents...
  • Across 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions.
  • Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, prompts, predictions, and evaluation scripts are open sourced at https://github.com/wzk1015/WorldCupArena.