evaluation framework
7 notes, newest first.
When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences
The researchers propose a framework to analyze how large language models (LLMs) handle conflicting specifications when integrating multiple sources of information. They compare four types of representation: pure…
Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
Researchers have developed an open-source framework to evaluate the effectiveness of AI agents powered by open-weight large language models (LLMs) in data preparation tasks for longitudinal population studies. The…
Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
The study introduces an open-source framework for evaluating the effectiveness of AI agents powered by open-weight large language models (LLMs) in data preparation tasks, particularly for longitudinal population…
LLMs and Agentic AI Systems for Smart Grids: A Tutorial on Architectures and Applications
The paper discusses the use of Large Language Models (LLMs) and Agentic AI systems in smart grids for tasks like forecasting, optimization, and control. It presents a design principle to ensure numerical results come…
LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos
Researchers introduce Elenchos, a framework to evaluate abductive reasoning (inferring latent hypotheses) of Large Language Models (LLMs). The study finds that LLMs can often detect changes in formal systems but…
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
The paper presents AgentGym2, a new evaluation framework for language models (LLMs) that are increasingly being used in real-world production environments. Most existing benchmarks evaluate these models in simplified…
Measuring the Gap Between Human and LLM Research Ideas
A study comparing the research ideas generated by large language models (LLMs) with those of human researchers. The authors build an evaluation framework to measure the gap between them using a two-axis taxonomy to…
