Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
56 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on benchmark
56 notes, page 2 of 3
VoLN: Vision-Only Long-Horizon Navigation---Paradigm, Benchmark, and Method
The text discusses a new navigation system called Vision-Only Long-Horizon Navigation (VoLN) for autonomous agents. In this system, the agent navigates using visual cues instead of relying on route-level instructions…
Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
The paper presents a new benchmark for testing the security of large language model (LLM) agents called 'Adaptive Adversaries'. The benchmark involves 21 scenarios where an adaptive attacker can observe prior defender…
Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
The paper presents a new benchmark for testing the security of LLM (Language Model) agents against adaptive adversaries in multi-turn interactions. The benchmark, called Adaptive Adversaries, evaluates how defenders…
The Shared Discovery Paradox: How a One-Answer Rule Turns Better Information into Worse Search
The study introduces a benchmark to analyze the Shared Discovery Paradox, a phenomenon where pooling information for one ranking leads to improved beliefs but reduced coverage in a discovery problem. The paradox arises…
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents in predicting football matches. The benchmark was used for the 2026 FIFA World Cup, but can be applied to…
MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
MedFailBench is a new open-source benchmark for evaluating the safety of medical AI systems, created by clinicians. It focuses on identifying where a system has failed in terms of safety boundaries rather than just…
Symbal: Detecting Systematic Misalignments in Model-Generated Captions
Researchers have developed Symbal, a method for detecting systematic misalignments in image-text pairs generated by multimodal large language models (MLLMs). These misalignments occur when the model consistently…
Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
The text discusses a new proactive assistance system named Vinci2, which provides assistance in continuous egocentric videos by considering the user's history, current activity, and whether help would be welcome. The…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
The text introduces IdeaGene-Bench (IG-Bench), a benchmark for evaluating AI systems' ability to follow the inheritance structure of scientific ideas, similar to how biological genomes evolve. IG-Bench uses the…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
A new benchmark called IdeaGene-Bench (IG-Bench) has been proposed for evaluating AI systems' ability to follow the inheritance structure of scientific ideas, similar to how biological genomes pass traits from one…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
The text presents IdeaGene-Bench (IG-Bench), a new benchmark for evaluating AI systems' ability to follow the structure of scientific idea inheritance. IG-Bench is based on the IdeaGene framework, which represents each…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The authors propose SolarChain-Eval, a benchmark for evaluating the trustworthiness of economic agents in decentralized energy markets. This benchmark ensures that agents make hourly decisions based on a…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The paper proposes SolarChain-Eval, a benchmark to evaluate the trustworthiness of economic agents in decentralized energy markets using a physics-constrained Markov Decision Process. The benchmark evaluates agents'…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The authors propose SolarChain-Eval, a benchmark for evaluating the trustworthiness of autonomous agents in decentralized energy markets. It assesses market utility, physical safety, slippage, action smoothness…
MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
The researchers introduce MIRA-Math, a benchmark for testing AI models' ability to ask for and integrate missing facts to solve mathematical problems. The benchmark contains 2310 instances from 22 mathematical families.
LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
This research investigates the performance of large language models (LLMs) in collaborative decision-making tasks where information is not fully available to all agents. The study introduces a benchmark for such tasks…
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
The paper introduces RuBench 1.0, a new benchmark for coding agents that uses tasks specified in Russian rather than English. The benchmark is designed to measure the performance of coding agents when they are given…
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Researchers have developed Spider 2.0-AIFunc, a benchmark for testing the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark contains 465 verified instances across…
ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
The paper introduces ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies in four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. The…
EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
Researchers introduce EvoAgentBench, a new benchmark for evaluating agent self-evolution in long-term LLM systems. The benchmark focuses on procedural reuse of experience, which includes searching, debugging, and…
Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas
Researchers introduce DramaSR-532K, a large-scale benchmark for speaker recognition in long-form TV dramas. They also propose DramaSR-LRM, a robust approach that uses a large reasoning model to improve speaker…
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
Researchers have developed TestEvo-Bench, a live benchmark for testing software tests and code co-evolution. The benchmark includes two tracks: test generation and test update, where an agent should either write new…
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
The paper introduces TestEvo-Bench, a new benchmark for evaluating test automation agents that can generate or update tests in response to code changes. The benchmark includes 746 test generation and 509 test update…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The paper introduces Adversarial Pragmatics, a benchmark and annotation protocol for evaluating language models' behavior in situations of instruction conflict, embedded commands, and policy ambiguity. The benchmark is…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
