Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
165 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on evaluation
165 notes, page 5 of 7
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
The paper introduces a reference-free framework to evaluate the quality of task-oriented conversational agent benchmarks using LLM (Large Language Model) judges. The framework assesses consistency, complexity, and…
Challenges in Evaluating Explanation Methods for Static and Evolving Data
The paper discusses the limitations in evaluating Explainable Artificial Intelligence (XAI) systems, specifically focusing on the DetoxAI image recognition system. It highlights the need for better evaluation methods…
Challenges in Evaluating Explanation Methods for Static and Evolving Data
This research paper highlights the issues in evaluating Explanation Methods for AI systems, using the DetoxAI image recognition system as an example. It discusses methods for adapting explanations to changing data…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
The paper introduces HarnessOpt-Bench, a benchmark for evaluating the ability of Language Models (LLMs) to optimize the harnesses surrounding them, which includes prompts, tools, control flow, memory, and orchestration…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers introduce HarnessOpt-Bench, a benchmark for evaluating the performance of Large Language Models (LLMs) in optimizing their own harnesses. The study examines five frontier LLMs on four tasks and finds that…
Learning Globally Reusable Skills for Coding Agents
Researchers propose a new method called GSE (Globalized Skill Evolution) to help Large Language Model agents continuously improve their skills without needing expensive retraining. The proposed method optimizes skill…
Tytan: Interactive Neurosymbolic Construction of Analytic Semantic Schemas from Relational Data
The paper presents Tytan, a system that automatically constructs an analytic semantic schema from relational databases to improve data analysis tools. The system combines symbolic analysis and LLM-based semantic…
Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning
The paper introduces CoCoEvolve, a method aimed at improving the consistency between chart, table, and code representations in AI systems. The method optimizes models based on agreement between different…
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
The text discusses a new benchmark called RepoProbe for evaluating the ability of Large Language Models (LLMs) to understand repository-scale code. The benchmark uses open-ended questions from GitHub Discussions and…
EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents
The paper introduces EMBL AI Librarian, a tool that allows AI agents to access and process life sciences literature more effectively by asking natural language questions and receiving relevant evidence in response. The…
From Textual Requirements to Microservice Architectures - A Comprehensive Evaluation of LLM-Based Design Synthesis
Researchers evaluated the ability of a Large Language Model (OpenAI o3) to generate microservice architectures based on textual requirements. The study found that the LLM was more effective when given examples…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The study presents ORCA-bench, a benchmark for evaluating the performance of large language models in oncall root cause analysis (RCA). The benchmark exposes a microservice system with six days of data and 1,079 RCA…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents' trajectories. The study finds that state-of-the-art VLMs have a…
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
Researchers have developed APS-RAG, a platform that helps scientists at the Advanced Photon Source (APS) access institutional knowledge through natural language queries. The platform uses a retrieval engine that fuses…
Evaluating RAG for French immigration law: a benchmark and baseline study
The paper presents a new benchmark for evaluating AI models used in French immigration law, focusing on permit-type recommendation, required document retrieval, and legal citation coverage. The study compares a…
Gubernaut: A Deterministic Homeostatic Controller for Affect-Regulated LLM Agents, Validated Across Independent Model Families
The research presents a new model-agnostic controller, Gubernaut Cognitive Controller (GCC), designed to control the behavior of Large Language Model (LLM) agents. The GCC aims to mitigate reactive failure modes in…
LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports
LLM-SoccerArena is a new platform that allows for the prospective evaluation of large language models' abilities to forecast real-world sports events before their outcomes are known. The platform provides a live…
Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
Researchers propose a method to evaluate large language models used as social simulators, such as synthetic survey respondents. They argue that simply checking if the simulated outcomes match human outcomes is not…
GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG
The paper introduces GRADRAG, a framework for improving Retrieval-Augmented Generation (RAG) systems by coordinating improvements across different components of the pipeline. It uses feedback from an Evaluator to…
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
The paper introduces a new benchmark called ICAE-Bench for evaluating coding agents in interactive project building settings. This benchmark aims to test the ability of agents to transform incomplete product intent…
Co-Evolving LLM Evaluators and Policies via DynamicRubric
The researchers propose DynamicRubric, a method that allows large language models to improve by co-evolving evaluators and policies. This method generates weighted binary rubrics for each candidate set and aggregates…
Human Grounded Evaluation of Large Language Models for Optical Network Automation
The paper introduces a new evaluation pipeline named HuGLEN for comparing large language models (LLMs) used in network automation. The pipeline uses an LLM to evaluate other LLMs and rates them based on a quality…
Stress Testing Concept Erasure with Large Language Model Agents
The paper introduces a new framework called Stress Testing Agents for Concept Erasure (STACE). STACE uses multiple Large Language Model (LLM) agents to stress-test models that have had certain concepts erased from…
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents in predicting football matches. The benchmark was used for the 2026 FIFA World Cup, but can be applied to…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
