Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
43 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on benchmarking
43 notes, page 2 of 2
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
OmniaBench is a new benchmark for evaluating the capabilities of general AI agents across diverse scenarios. It includes over 1,400 tasks with a hierarchical taxonomy of 90 level-1 and 354 level-2 domains based on app…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
Researchers have introduced OmniaBench, a new benchmark for evaluating the abilities of general AI agents across various scenarios with explicit state spaces. The benchmark includes a hierarchical taxonomy covering…
AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking
The paper presents AutoMatBench, an automatic optimization toolkit that addresses the limitations of MatBench, a widely used material property prediction benchmarking tool. MatBench lacks the ability to reflect the…
VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion
The paper introduces VoxENES 2026, a new benchmark for testing speech synthesis systems against modern text-to-speech (TTS) and voice conversion (VC) methods. The benchmark includes 53,628 audio samples in English and…
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Researchers introduce 'blind-spots-bench', a new benchmark to evaluate and expose persistent blind spots in modern AI models, especially those from closed-source providers, which can outperform open-weight models by up…
OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice
The paper introduces OmniFood-Bench, a new benchmark for evaluating Large Vision-Language Models (VLMs) in the context of personalized healthcare and dietary management. The benchmark is designed to assess the ability…
OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice
The paper introduces OmniFood-Bench, a comprehensive benchmark for evaluating the performance of Vision-Language Models (VLMs) in personalized healthcare and dietary management. The benchmark focuses on three…
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Researchers introduce a new benchmark called Spider 2.0-AIFunc to test the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark includes 465 verified instances…
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
Researchers have introduced UI2App, a new benchmark focused on evaluating the ability of large language models to recover application behavior from screenshots alone, without any textual or behavioral guidance. The…
VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection
The paper introduces VendorBench-100, a benchmark for evaluating deepfake image detection models across three paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. The…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The text introduces a new benchmark called SovereignPA-Bench for evaluating user-owned personal agents. The benchmark focuses on aspects like preserving user sovereignty, privacy, consent, and resisting manipulative…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The text introduces 'adversarial pragmatics', a new benchmark and annotation protocol to evaluate the behavior of language models in various ambiguous situations. The goal is to clarify whether failures occur due to…
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
The paper evaluates three performance-optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to assess coding agents. The authors find that these benchmarks have issues with stability, scoring rules, and the…
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
The authors propose MemSyco-Bench, a new benchmark to evaluate how memory affects decision-making in language models (LLMs) used in AI agents. The benchmark aims to assess if the agents can handle issues of sycophancy…
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Researchers have proposed a new benchmark called MemSyco-Bench to evaluate how memory in large language models (LLMs) can lead to sycophancy, where agents over-align with the user at the cost of factual accuracy or…
AxDafny: Agentic Verified Code Generation in Dafny
The paper presents AxDafny, a framework for generating executable code and verification artifacts in Dafny using a verifier-guided repair approach. The framework significantly improves verification success over the…
NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
Researchers have created a new benchmark called NuclearQAv2 to evaluate the performance of large language models (LLMs) in the domain of nuclear engineering. The benchmark consists of approximately 1,240…
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
The study introduces Argus, a benchmark for post-hoc uncertainty quantification (UQ) in single-step executable GUI grounding across various vision-language models and datasets. The research compares 27 methods of UQ…
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
Researchers have introduced a new benchmark called EHR-Complex to test the performance of clinical agents that interact with electronic health records (EHRs). The benchmark is built on a large dataset from MIMIC-IV and…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
