An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Hao He, Xueying Liu, Chris J. Kuhlman, Xinwei Deng
• Venue: arXiv: stat.ME
• Published: 2026-07-07
• Collected: 2026-07-08
• URL: http://arxiv.org/abs/2607.06413v1
• PDF: https://arxiv.org/pdf/2607.06413v1
• DOI: UNKNOWN

Plain-Language Brief

The researchers propose an experimental design framework to systematically evaluate the autonomous model discovery behavior of large language model coding agents, like Codex and Claude Code. The framework considers factors such as reasoning effort, task, optimization metric, and composition of training data to quantify variability and identify important factors.

What the Document Says

• The text discusses an experimental design approach for evaluating autonomous model discovery in large language model coding agents.
• Two specific agents, Codex and Claude Code, are investigated under controlled experimental factors.
• Factors include agent's reasoning effort, task, optimization metric, and composition of training data.
• For each agent-task-metric combination, regression models and inference are conducted for various outcomes like output quality, cost, time, and process complexity.
• A utility-aligned canonical decomposition is developed to characterize the dominant direction of reasoning effort's impact and assess its alignment with a performance-cost utility direction.

Key Concepts Mentioned

• autonomous model discovery
• large language model coding agents
• Codex
• Claude Code
• experimental design framework
• reasoning effort
• task
• optimization metric
• composition of training data
• regression models
• utility-aligned canonical decomposition

Claims Made by the Document

• The proposed framework can systematically evaluate the autonomous model discovery process of large language model coding agents.

Evidence Used

• The text describes a framework for evaluating the behavior of these agents and mentions the specific factors it considers.

Methods / Approach

• Investigation under controlled experimental factors, regression models, inference

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language model coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run. In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors. The proposed framework treats these agents as stochastic model-discovery operators, which map task-specific discovery data and an optimization target to a fitted model. Specifically, we investigate two such operators, Codex and Claude Code, under controlled experimental factors including agent's reasoning effort, task, optimization metric, and composition of training data. For each agent-task-metric combination, regression models and inference are conducted for multiple responses such as output quality, dollar cost, wall-clock time, and process complexity. Furthermore, we develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect and to assess whether that direction aligns with a performance-cost utility direction. The proposed framework is demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.