Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Mack Nixon, Liam Wright, Yevgeniya Kovalchuk, Alison Fang-Wei Wu, Martin Danka, Andy Boyd, David Bann
• Venue: arXiv: cs.AI
• Published: 2026-07-23
• Collected: 2026-07-24
• URL: http://arxiv.org/abs/2607.21482v1
• PDF: https://arxiv.org/pdf/2607.21482v1
• DOI: UNKNOWN

Plain-Language Brief

The study introduces an open-source framework for evaluating the effectiveness of AI agents powered by open-weight large language models (LLMs) in data preparation tasks, particularly for longitudinal population studies. The framework includes a curated ground-truth dataset, task definitions, and automated routines for evaluating LLM-produced R code. The study finds that state-of-the-art, 31-35B parameter models show high efficacy (up to 87.9% average task completion) in 20 data preparation tasks. The performance of open-weight LLMs running on consumer-grade hardware suggests a promising path for AI-assisted data preparation in governance-restricted research settings.

What the Document Says

• The study presents an open-source framework for evaluating AI agents using open-weight LLMs for data preparation tasks in longitudinal population studies.
• The framework includes a curated ground-truth dataset, task definitions, and automated routines for evaluating LLM-produced R code.
• State-of-the-art, 31-35B parameter models have high efficacy (up to 87.9% average task completion) in 20 data preparation tasks.
• The performance of open-weight LLMs running on consumer-grade hardware shows promise for AI-assisted data preparation in governance-restricted research settings.
• The framework is available at https://github.com/UCL-ARC/RRBench.

Key Concepts Mentioned

• large language models
• AI agents
• open-weight models
• data preparation
• longitudinal population studies
• British cohort study
• R code
• consumer-grade hardware

Claims Made by the Document

• State-of-the-art, 31-35B parameter models have high efficacy in data preparation tasks.
• The performance of open-weight LLMs running on consumer-grade hardware shows promise for AI-assisted data preparation.

Evidence Used

• The study benchmarked the performance of LLMs across different deployment spectra and found high efficacy (up to 87.9% average task completion) in 20 data preparation tasks.
• The study cites the promise shown by the performance of open-weight LLMs running on consumer-grade hardware.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ('average task completion' up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: https://github.com/UCL-ARC/RRBench.