Measuring the Gap Between Human and LLM Research Ideas

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Ziyu Chen, Yilun Zhao, Arman Cohan
• Venue: arXiv: cs.CL
• Published: 2026-07-01
• Collected: 2026-07-02
• URL: http://arxiv.org/abs/2607.01233v1
• PDF: https://arxiv.org/pdf/2607.01233v1
• DOI: UNKNOWN

Plain-Language Brief

A study comparing the research ideas generated by large language models (LLMs) with those of human researchers. The authors build an evaluation framework to measure the gap between them using a two-axis taxonomy to profile each idea.

What the Document Says

• The paper discusses the use of LLMs for generating research ideas and evaluates their performance compared to human researchers.
• An evaluation framework is built based on high-quality human research papers.
• Paper titles and summaries are used to generate new ideas using LLMs.
• A two-axis research-taste taxonomy is introduced for profiling each idea by opportunity pattern and research paradigm.
• The study finds a consistent distributional gap between the LLM-generated ideas and human paper reference distribution.
• LLM ideas are concentrated around bridge-like opportunities and synthesis methods, while human ideas spread more broadly.

Key Concepts Mentioned

• large language models (LLMs)
• evaluation framework
• high-quality human research papers
• two-axis research-taste taxonomy
• opportunity pattern
• research paradigm

Claims Made by the Document

• LLM-generated ideas differ from human researchers' ideas.
• The LLM idea distribution is narrower and systematically shifted compared to human research taste.

Evidence Used

• Observed distributional gap in the generated ideas across different LLMs.
• LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, while human ideas spread more broadly.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.