Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
56 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on benchmark
56 notes, page 1 of 3
ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
The preprint builds a benchmark from judgments by 184 lead authors of 207 recent computer-science papers, who identified prior papers that advanced or could have advanced their projects and provided rationales. The…
WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents
The preprint presents 213 anomaly tasks across 13 Unreal Engine 5 and three.js environments, covering five anomaly families. It evaluates five frontier models under a fixed exploration budget using VLA-led exploration…
Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
The preprint describes a fully synthetic, fact-grounded longitudinal EHR benchmark built from public medical education material, with no PHI. The abstract reports 1,268 patients and 5,602 encounters, ICD-10-CM/SNOMED…
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
The paper presents AI4AI-Bench, a benchmark for evaluating Large Language Model (LLM) agents in recursive self-improvement. The goal is to determine if an AI system can improve the process that produces future AI…
ContractScrub: A benchmark for final review of legal contracts
The paper introduces ContractScrub, a new benchmark for evaluating the performance of Language Models (LLMs) in reviewing legal contracts for errors. The study finds that while LLMs perform well on general benchmarks…
DeepWeaver: Bridging the Evidence Synthesis Gap in Open-Ended Question Answering
The paper introduces DeepWeaver, a framework designed to address the gap between evidence retrieval and generation in open-ended question answering systems. DeepWeaver organizes and synthesizes noisy and fragmented…
Chronocooked: A Benchmark for Implicit Interval Timing in Reinforcement Learning Agents
The paper introduces Chronocooked, a benchmark suite for studying how reinforcement learning (RL) agents make decisions over time. It's inspired by Overcooked and includes cooking scenarios that require temporal…
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
The paper introduces CTBench, a public benchmark for evaluating AI agents' ability to perform troubleshooting tasks in telecom network operations. The focus is on root cause analysis and path restoration. The authors…
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
The researchers introduce VAKRA, a benchmark for evaluating the ability of agents to reason across structured APIs and document collections in enterprise settings. The benchmark includes over 8,000 executable APIs…
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
The paper introduces VICBench, a multi-language benchmark for code vulnerability detection. It was created by human experts and an agentic workflow, featuring 100 verified vulnerabilities across 88 projects in Python…
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Researchers have created a new framework called V-FiLLM to evaluate the abilities of Language Models (LLMs) in financial reasoning using structured data. The framework generates questions from executable computation…
Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
Researchers introduce Avalon-ToM-Bench, a new benchmark for evaluating fine-grained Theory of Mind (understanding others' thoughts and motivations) in AI models. The benchmark uses the asymmetric-information mechanics…
Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
A new benchmark called Avalon-ToM-Bench is introduced to evaluate fine-grained Theory of Mind (ToM) abilities in agents. The benchmark operates through the asymmetric-information mechanics of a game named The…
DSLE: A Learning Environment for Dark Souls Boss Encounters
Researchers have created a learning environment for AI agents called DSLE, which is based on the game Dark Souls: Remastered. The environment includes 22 boss encounters that can be used as benchmarks. The researchers…
FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows
Researchers have created a new benchmark for self-evolving agents in financial workflows called FinEvo-Bench. The benchmark includes 120 real-life tasks across six domains, with each task having a specific professional…
FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows
The research introduces a new benchmark called FinEvo-Bench for evaluating the performance of self-evolving agents in professional financial workflows. The benchmark consists of 120 real-case-grounded tasks across six…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
Researchers introduce ORCA-bench, a new benchmark that tests the ability of large language models to perform root cause analysis (RCA) in a production-like setting. The benchmark includes a microservice system with six…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The paper introduces ORCA-bench, a benchmark for testing the performance of large language models in oncall root cause analysis (RCA). The benchmark includes a live microservice system with six days of metrics, logs…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The study presents ORCA-bench, a benchmark for evaluating the performance of large language models in oncall root cause analysis (RCA). The benchmark exposes a microservice system with six days of data and 1,079 RCA…
MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair
Researchers have developed a benchmark called MemSecBench to evaluate the lifecycle security of agent memory systems. The benchmark contains 310 cases from various contexts, each following a specific protocol and using…
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Researchers have developed a new benchmark called OmegaUse-OfficeVal to test the performance of large language model (LLM) agents in completing long-horizon office-suite tasks. The benchmark includes 100 tasks that…
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
The study introduces SecRespond, a new benchmark for evaluating AI agents in post-compromise incident response scenarios. The benchmark tests the ability of agents to produce forensic reports and remediation plans…
Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
The researchers propose a benchmark called Setoka to evaluate how well personalized agents understand users' abstract characteristics, such as personality traits and behaviors, beyond just retrieving explicit facts…
RUMBA: Russian User Memory Benchmark
The authors introduce RUMBA (Russian User Memory Benchmark), a new long-term conversational memory benchmark for Russian. It offers a detailed taxonomy of memory-centric question types and a unified methodology that…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
