ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Source facts

The preprint builds a benchmark from judgments by 184 lead authors of 207 recent computer-science papers, who identified prior papers that advanced or could have advanced their projects and provided rationales. The abstract reports that agentic search did not outperform embedding retrieval on the task and that even a model with possible exposure to completed papers reached only 0.51 Recall@20.

Analysis

ScholarCatalyst targets a difficult layer of scientific assistance: finding ideas that are useful, not merely documents that are semantically similar. The gap between ordinary retrieval and expert-guided research retrieval suggests that scientific agents need better training signals for relevance, novelty, and problem-specific intuition.

Uncertainty/Doubt

• This is an arXiv preprint. The author annotations, temporal retrieval setup, and reported recall results were not independently audited here.