Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
165 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on evaluation
165 notes, page 6 of 7
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
The researchers evaluated language model security agents based on cost efficiency in addition to their success rates, focusing on offensive CTF challenges and defensive Splunk BOTS v1 investigation challenges. They…
Concept-Guided Spatial Regularization for World Models in Atari Pong
Researchers evaluate five world model agents in Atari Pong and find inconsistencies and failures in the generated video trajectories. They hypothesize that insufficient modeling of task-critical concepts may contribute…
Concept-Guided Spatial Regularization for World Models in Atari Pong
Researchers evaluate five world model agents in Atari Pong and find failures in the generated video trajectories, such as ball disappearance and incorrect motion. They propose a new method called Concept-Guided Spatial…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Researchers introduce MM-IssueLoc, a benchmark and evaluation protocol designed specifically for repository-level issue localization with visual evidence. The dataset includes 652 issue-PR instances across multiple…
Plover: Steering GUI Agents through Plan-Centric Interaction
The paper presents Plover, a system for controlling graphical user interface (GUI) automation agents in real-world environments. It makes planning and adaptation visible to users, allowing them to supervise, correct…
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
The text is about AgentCompass, an open-source evaluation infrastructure designed to evaluate Large Language Models (LLMs) as they evolve into autonomous agents. The infrastructure aims to address the current…
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
The research evaluates if improvements from agent-optimization methods can stack up over time in a continual learning setting. Three optimizer-driven methods (GEPA, Meta Harness, and RELAI's Verifiable Continual…
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
Researchers investigated if improvements made by agent-optimization methods persist when agents are optimized again on new tasks. They found that one method called RELAI-VCL was the only one that both transferred…
MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models
The research paper introduces MAGIC, a system that generates multi-scene game worlds using large language models (LLMs). The system overcomes three obstacles in single-scene methods: cross-scene consistency, in-scene…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
The paper introduces MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. It supports multi-image, multi-turn tasks and offers a stateful execution environment with over 500 tools across 16…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Researchers introduce MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. This framework supports over 500 tools across 16 application domains and can handle multi-image, multi-turn tasks with…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Researchers have introduced MM-ToolSandBox, a tool for evaluating visual agents that can perform tasks by calling tools based on visual inputs. The framework includes over 500 tools across 16 application domains and…
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
A study finds that large language models often struggle with two types of errors: answering questions incorrectly and attempting to answer unanswerable or misleading questions. The researchers propose a method to…
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
The study investigates the inconsistency in Language Model (LLM) as judge scores when different evaluators are used. It compares two methods of upgrading LLMs: scaling Qwen3 dense judges and moving across MiniMax APIs…
Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors
The research discusses the study of multi-agent AI control in real-world scenarios where multiple agents operate over shared infrastructure. The paper introduces FakeLab, a synthetic AI-lab codebase used to evaluate…
Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
The paper introduces a method called Reasoning Consistency Scanning to assess the logical consistency between stated reasoning and answers in AI safety evaluation transcripts. The method can be applied to various AI…
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
The researchers evaluate the performance of uncertainty estimation methods across 22 languages in high-, mid-, and low-resource settings using two human-curated question and answer datasets. They compare open and…
From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
A study presents a framework for improving the accuracy and reliability of biomedical question answering by using a large language model (LLM) that adapts its inference strategies based on the type of question (yes/no…
LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
This research investigates the performance of large language models (LLMs) in collaborative decision-making tasks where information is not fully available to all agents. The study introduces a benchmark for such tasks…
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Researchers have developed Spider 2.0-AIFunc, a benchmark for testing the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark contains 465 verified instances across…
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
The paper presents AgentGym2, a new evaluation framework designed to test language models in realistic, complex scenarios that mimic real-world deployments. It highlights that most existing benchmarks do not fully…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The text introduces a new benchmark called SovereignPA-Bench for evaluating user-owned personal agents. The benchmark focuses on aspects like preserving user sovereignty, privacy, consent, and resisting manipulative…
EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
A new evaluation setting called Autonomous Policy Evolution (APE) is introduced in a benchmark named EvoPolicyGym, which is used to test how autonomous agents iteratively improve their policy systems within a fixed…
EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
The paper introduces EvoPolicyGym, a new benchmark for evaluating how autonomous agents improve their executable policies through feedback in interactive environments. The authors use this benchmark to evaluate GPT-5.5…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
