Can AI agents conduct open-ended AI research? Early evidence from two case studies

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, Arvind Narayanan
• Venue: arXiv: cs.AI
• Published: 2026-07-29
• Collected: 2026-07-30
• URL: http://arxiv.org/abs/2607.27191v1
• PDF: https://arxiv.org/pdf/2607.27191v1
• DOI: UNKNOWN

Plain-Language Brief

In a study, two unpublished AI research papers were given to AI agents for completion. Despite the agents completing the engineering tasks without human help, they failed to make significant progress towards answering the research questions. The authors identified five recurring failure modes.

What the Document Says

• The paper discusses the lack of evidence on whether AI agents can conduct open-ended AI research.
• It introduces the concept of 'shadow evaluations' where an agent takes on the central research question of an unpublished paper and the paper's original authors grade its output.
• Two unpublished NeurIPS 2026 submissions were subjected to shadow evaluations, with AI agents given six days and thousands of dollars of compute.
• The agents completed the engineering tasks without human help but failed to make substantial progress towards answering the research questions.
• Both papers were unambiguously rejected by the authors as a result.
• Five recurring failure modes were identified: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift.
• A robustness check with a second model and scaffold reproduced these failures.
• The study concludes that today's agents can do the engineering of AI research but struggle with critical parts of the research lifecycle.

Key Concepts Mentioned

• AI agents
• open-ended AI research
• shadow evaluations
• NeurIPS
• research questions
• failure modes

Claims Made by the Document

• Today's AI agents can do the engineering of AI research.
• AI agents struggle with critical parts of the research lifecycle.

Evidence Used

• The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions.
• The study results provide early evidence that today's agents can do the engineering of AI research but struggle with critical parts of the research lifecycle.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.