reproducibility
4 notes, newest first.
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
The preprint evaluates agents as configurable systems whose task information, reasoning settings, self-verification, time budget, and backbone model all affect outcomes. On four scientific tasks, the abstract reports…
Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation
The protocol proposes three actions: refuse when a clean reference stream or matched runtime comparison lacks support; decompose execution, false admission, and structural hypotheses instead of collapsing them into…
AI-Assisted Computational Reproducibility on the FABRIC Testbed
This paper discusses the use of an international research testbed, FABRIC, combined with a large language model (LLM) coding assistant called LoomAI, to simplify the reproduction of published experiments across various…
AI-Assisted Computational Reproducibility on the FABRIC Testbed
This paper discusses how the FABRIC testbed, along with an AI coding assistant named LoomAI, can help simplify reproducing published experiments across various domains. The authors present three case studies and…
