Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

Source

arXiv

arxiv.org/abs/2607.21482v1

Read the source

Source

Plain-Language Brief

Researchers introduce an open-source framework for evaluating the effectiveness of AI agents powered by open-weight large language models (LLMs) in data preparation tasks, specifically for longitudinal population studies. The framework includes a curated ground-truth dataset and task definitions for various data preparation tasks. They benchmarked LLMs on 20 tasks with consumer-grade hardware, showing that state-of-the-art models perform well but run on open-weight LLMs have potential in governance-restricted research settings.

What the Document Says

  • The text discusses the use of large language models (LLMs) and agents for code development, with a focus on their adoption in research using personal data that is subject to governance requirements limiting data transmission to external services.
  • It mentions locally deployable open-weight models as an alternative, as they keep sensitive data within the local environment.
  • The researchers introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs in data preparation tasks for longitudinal population studies.
  • The framework includes a curated ground-truth dataset and task definitions for various data preparation tasks such as category harmonization and multi-wave merging.
  • They also mention automated routines for evaluating the LLM-produced R code and outputted data.
  • The text states that they benchmarked LLMs across different deployment spectra, with consumer-grade hardware being one of them.
  • The current state-of-the-art, 31-35B parameter models were found to almost saturate their benchmark, achieving up to 87.9% 'average task completion'.
  • Lastly, they conclude that the performance of open-weight LLMs running on consumer-grade hardware shows promise for AI-assisted data preparation in governance-restricted research settings.
  • The framework is publicly available at: https://github.com/UCL-ARC/RRBench.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • AI agents
  • Data preparation
  • Longitudinal population studies
  • Governance requirements
  • Open-weight models
  • Consumer-grade hardware

Claims Made by the Document

  • Open-weight LLMs have potential in governance-restricted research settings.
  • Current state-of-the-art, 31-35B parameter models perform well on their benchmark.

Evidence Used

  • Performance of open-weight LLMs mentioned in conclusion
  • Benchmark results of state-of-the-art LLMs provided

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ('average task completion' up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: https://github.com/UCL-ARC/RRBench.