Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shaolong Chen, Yanlin Fei, Nazhou Liu, Xinmiao Yu, Lei Li, Rahul Thapa, Madalina Ciobanu, Qingqing Mao, Ritankar Das
• Venue: arXiv: cs.AI
• Published: 2026-08-17
• Collected: 2026-08-18
• URL: http://arxiv.org/abs/2608.16645v1
• PDF: https://arxiv.org/pdf/2608.16645v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper introduces Reconstruction, a blind benchmark for testing language models' ability to recover the true research idea of a published paper using only its pre-publication bibliography. The authors use an anti-leakage protocol to prevent models from gaining information about the seed paper during the evaluation process. Seven frontier models were evaluated across six scientific domains, achieving modest Match rates (approximately 3-15%). A reference-only multi-agent pipeline was also developed and tested, resulting in higher Match rates of approximately 23-42%.

What the Document Says

• The paper introduces a benchmark called Reconstruction to test language models' ability to recover the true research idea of a published paper from its pre-publication bibliography. The benchmark withholds the seed paper and all contemporaneous or future literature, requiring models to propose hypotheses that an independent large language model judges against the held-out ground-truth idea.
• The authors use an anti-leakage protocol that includes a temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies to prevent prompt-time leakage of the seed idea.
• Six scientific domains and 643 papers were evaluated across seven frontier models, resulting in Match rates approximately between 3% and 15%.
• The paper also presents a reference-only multi-agent pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search.

Key Concepts Mentioned

• language model
• research idea recovery
• blind benchmark
• pre-publication bibliography
• anti-leakage protocol

Claims Made by the Document

• Language models can recover the true research idea of a published paper when given only its pre-publication bibliography.

Evidence Used

• The introduction and description of Reconstruction as a benchmark for evaluating language models' ability to recover research ideas.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.