Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
21 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on ai evaluation
21 notes
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
The paper introduces Graph-Structured Rubrics (GSR), a method that converts rubrics into response-independent typed evaluation graphs before observing responses. This allows for more efficient and accurate scoring of…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers have developed a new method to assess the quality of benchmarks used to evaluate task-oriented conversational agents. The method uses artificial intelligence judges to determine if benchmarks are…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
The authors present a new framework to evaluate the quality of benchmarks used to test conversational agents. The framework uses language models (LLMs) as judges to assess the consistency, complexity, and policy…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of Vision-Language Models (VLMs) that judge Computer-Using Agents' (CUAs) trajectories. The study finds that even state-of-the-art VLMs…
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Two case studies show that current AI agents struggle to make substantial progress towards answering open-ended research questions in AI, even with six days and significant computing resources.
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluated six large language models on their ability to understand scientific visualizations. The study used a standardized test with 49 items based on 18 visualizations and illustrations, covering eight…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluated six multimodal large language models (MLLMs) on a standardized scientific visualization literacy assessment test, consisting of 49 items based on 18 visualizations and illustrations. The study…
Can We Trust Item Response Theory for AI Evaluation?
This research examines the reliability of using Item Response Theory (IRT) for evaluating AI performance in benchmarks. The study finds that classical estimators may become infeasible in large benchmark settings, while…
Can We Trust Item Response Theory for AI Evaluation?
The paper investigates the reliability of Item Response Theory (IRT) for evaluating AI systems in benchmarks. The study finds that traditional IRT estimation tools may not be suitable for AI benchmark data, which often…
Can We Trust Item Response Theory for AI Evaluation?
The paper explores the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI) systems in benchmarks. The authors find that standard IRT estimation tools may not be reliable in large…
Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models
The study introduces a methodology called PoPE (Popperian Placebo-controlled Evaluation) to measure whether a frozen small code language model (LLM) can use evidence that falsifies its own generated code. The study…
LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos
The study introduces Elenchos, a framework to evaluate the abductive reasoning capabilities of large language models (LLMs). The research finds that while LLMs can detect when changes have been made in a formal system…
Psychological Competence as a Missing Dimension in AI Evaluation
The paper discusses the need to consider psychological competence as an important aspect in evaluating AI systems that interact directly with humans, particularly in roles like advisors, coaches, tutors, and…
Psychological Competence as a Missing Dimension in AI Evaluation
The text discusses a need to evaluate human-AI interaction beyond just technical performance, focusing on how AI systems can affect users' cognition, emotions, decision-making, and behavior. The proposed evaluation…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The paper proposes SolarChain-Eval, a benchmark for evaluating the trustworthiness of economic agents in decentralized energy markets. The benchmark is designed to assess the performance of AI systems in cyber-physical…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The authors propose SolarChain-Eval, a benchmark for evaluating the trustworthiness of autonomous agents in decentralized energy markets. It assesses market utility, physical safety, slippage, action smoothness…
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
The paper investigates how changes in the evaluators used by large language models (LLMs) to score candidate responses can impact the model's performance and reliability. The study examines two common upgrade paths for…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The paper introduces SovereignPA-Bench, a benchmark for evaluating user-owned personal agents in terms of preserving user sovereignty under various conditions such as evolving intent, platform mediation, privacy…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The text discusses a new benchmark called SovereignPA-Bench for evaluating the performance of user-owned personal agents. These agents are designed to remember preferences, filter information, use tools, and negotiate…
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
The paper introduces AdversaBench, a red-teaming pipeline for evaluating large language models. The pipeline generates hard inputs using five structured operators and confirms failures through a three-judge panel with…
Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
The research evaluates automated graders for agentic data analysis systems using LAMBDA, a multi-agent system, on 153 numerical QRData tasks. The study introduces a three-layer human-AI grading cascade and an iterative…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
