AI evaluation and benchmarks
320 notes, newest first. Includes the vault tags evaluation, evaluation methods, ai evaluation, performance evaluation, benchmark, benchmarks, benchmarking, llm evaluation, large language model evaluation, agent evaluation.
Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs
Across 18 base-instruct model pairs spanning Qwen, Llama, and Gemma, the authors report that harmfulness is more readily decoded from attention updates while MLP updates contribute more strongly to harm-selective…
DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design
DRYLAB-Bench evaluates agentic mutation-design capability in 30 hazardous scenarios distilled from biosecurity guidelines, using multi-round tool-augmented interactions. It scores designs against 187,695 empirical…
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
The study tests whether scoring signals in on-policy self-distillation localize errors and push probability away from erroneous student tokens. It reports that full-distribution KL and token log-ratio signals do…
RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics
RATED combines a frozen dual encoder for semantic matching with a lightweight contextual-bandit module that reranks tools using real-time execution feedback. The authors introduce EcoAgent, a benchmark of 5,172 tools…
X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems
X-AgentRed constructs a directed graph of tool dependencies, effects, harms, and execution preconditions to discover harmful cross-service action paths. Execution feedback expands and refines the paths through tool…
BrickBench: Evaluating Agentic Brick Design
The preprint introduces BrickBench, a benchmark for text-conditioned LEGO-set design. Agents must produce assemblies that satisfy semantic and design criteria while remaining physically buildable, selecting parts from…
Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
The preprint evaluates epistemic humility through three trajectory-level behaviors: Identify, Solve, and Escalate. It tests controlled and naturally occurring conflicts between parametric knowledge and retrieved…
OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport
The preprint proposes OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent. It evaluates full-reference…
When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
The preprint proposes Reasoning Adaptation through Cross-Turn Estimation (RACE), which estimates when prior reasoning remains sufficient for subsequent actions. Its LoGiC procedure identifies reasoning turns whose…
Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff
The preprint develops a population-level model in which agent fitness depends on cybersecurity capability. It reports that without collaboration, takeoff requires individual capability above a critical threshold; with…
GeoReform: Reflective Formalization Evolution for Multimodal Geometry Problem Solving
The preprint proposes GeoReform, which treats geometric formalization as an optimizable policy rather than a fixed parser output. It collects failed rollouts, diagnoses representation defects, and mutates how entities…
Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
The preprint audits whether harmful refusal is a single measurable attribute in HELM Safety. It reports saturation in three candidate datasets and uses multidimensional item-response theory plus differential item…
Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness
The preprint tests whether legal models' named statutes or precedents causally support their verdicts. Holding case facts fixed, it substitutes the named authority and examines verdict changes and hidden-state dynamics…
On the estimation and validity of AI time horizons---a statistical look at the METR plot
The preprint recomputes METR-style 50% AI time horizons over 228 tasks and 26 AIs using splines and item-response theory. The authors report that the fitted spline is nearly flat from roughly 2–30 minutes and close to…
EvoAlloc: A Self-Evolving Resource Allocation Agent for Efficient Program Evolution
EvoAlloc is described as a self-evolving resource-allocation agent for LLM-based program evolution. It consolidates prior search and allocation outcomes into reusable experience, revises its allocation strategy, and…
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
The preprint frames multimodal synergy as higher-order dependence that can be lost by pairwise fusion. HRIL uses an empirical cross-moment tensor with Tucker decomposition and a synergy-aware regularizer. The authors…
Recursive Self-Improvement through Multi-Agent Self-Supervision
The work describes MASS, which alternates evolutionary workflow optimization with supervised fine-tuning on self-generated trajectories. The abstract reports discovery of multi-agent workflows and, over two cycles with…
Prior or Feedback? What an LLM Uses When Adapting Neural Operators
The study tests whether an LLM scientific agent adapts neural-operator tuning using experimental feedback under a limited trial budget. The authors report lower held-out nRMSE than random search and Bayesian…
Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems
The preprint tests regulatory-rule sensitivity by deleting, swapping, or negating a rule while holding the case fixed across five models and 20 domains. It reports that verdicts are often invariant to large rule…
ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences
The preprint formalizes fixed-parameter program self-evolution that links task solving, scientific verification, and program updates. ScienceClaw-Eval spans 23 disciplines and measures correctness, evolutionary gain…
Sherpa: Teaching LLMs to Teach Adaptively
The preprint introduces Sherpa, a multi-turn reinforcement-learning framework that trains an LLM teacher against multiple student archetypes with distinct learning preferences. The teacher is optimized directly against…
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
The preprint presents Transect, an open-source package built on Inspect Scout for analyzing long agent runs whose transcripts can span hundreds of pages. Configurable evaluation families align recorded events, token…
VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoning
The preprint proposes an agent harness that co-evolves the policy, training curriculum, and judge. A Policy Improvement Loop diagnoses recurring failures and builds an adaptive curriculum; when verification becomes a…
Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
The preprint introduces BOTTLED, a benchmark in which agents receive an unlabelled workload and must complete it under fixed time, compute, and LLM API budgets. Agents may train a small model or write a reusable…
MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation
The preprint presents MedZERO, a self-evolving framework for open-ended medical reasoning. An Examiner generates medical question-option pairs, while a Reasoner solves them with evidence-grounded multi-turn reasoning…
ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
The preprint introduces a benchmark for unified evaluation of risk-treatment capabilities in coding agents. It maps Avoidance, Transfer, Mitigation, and Acceptance from software risk management to 200…
SquidAgent: Parallelize Wisely, Coordinate Efficiently
The preprint identifies re-exploration and alignment as hidden costs that can make parallel multi-agent systems slower than a single-agent baseline. It proposes a token-based decision criterion, direct forking from the…
WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?
The preprint introduces WorldSolver, a benchmark of 168 simulation tasks derived from physical phenomena in 61 classic computer-graphics papers across seven physical domains. Each task provides a code scaffold while…
AgentPrivArena: Evaluating and Auditing Real-world AI Agent Privacy
The preprint introduces AgentPrivArena, a reproducible environment that integrates authentic MCP tools and self-hosted services to evaluate privacy risks during realistic multi-step agent execution. It proposes…
What Did the Agent Actually Do? Evidence-Grounded Oversight for Long-Horizon Agents
The preprint introduces AgentMonBench, a software-engineering benchmark for judging requirement/behavior alignment and identifying consequential autonomous decisions that warrant verification. Its Evidence-Grounded…
HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
The preprint introduces HERA, a framework that constructs verifiable feasible/infeasible task pairs through controlled environment mutations and adapts the agent harness toward prior weaknesses. The abstract reports…
ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement
The preprint describes an autonomous post-training harness that attributes errors, selects update directions, and executes model updates under a limited compute budget. It combines metric- and case-level analysis…
HazardWeaver: Scientific Route Selection for Hazard Analysis Agents
The preprint formulates hazard-analysis automation as state-dependent scientific route selection. HazardWeaver combines an evidence-linked Hazard Knowledge Compiler with a Hazard Capability Graph that checks…
HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
The preprint presents 423 manually authored and human-validated questions across 13 languages. Items require obscure evidence discovery, multi-step clue chains, and heterogeneous sources such as videos, scanned…
Recursive Harness Self-Improvement for Frontier Reasoning Data Synthesis
The preprint introduces task-harness co-evolution for recursive reasoning-data synthesis. Online self-improvement turns solver failures into reusable skills, while post-task updates revise skills, prompts, and…
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
The preprint evaluates agents as configurable systems whose task information, reasoning settings, self-verification, time budget, and backbone model all affect outcomes. On four scientific tasks, the abstract reports…
Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval
The preprint identifies a retrieval-level positivity problem: memories that are never retrieved cannot have their utility identified by store-level interventions. It proposes Causal Memory Policy, which reserves…
Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks
The preprint argues that many failures of 2B–9B open models on real tasks arise from the harness rather than the model. Mingbird is a local-first harness for Windows and Ollama with mechanisms including a byte-level…
ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
The preprint builds a benchmark from judgments by 184 lead authors of 207 recent computer-science papers, who identified prior papers that advanced or could have advanced their projects and provided rationales. The…
VISTA: A Visual Harness for Reasoning in an Interactive World
The technical report introduces a visual harness that gives a multimodal model long-horizon visual context through direct perception, lossless visual memory, active retrieval, and input reorganization. The abstract…
The AI Assessment Sandbox Configurator: A Framework to Support Technical Assessment in AI Regulatory Sandboxes
The preprint describes an open-source framework for technical testing in AI regulatory sandboxes, motivated by EU AI Act obligations. It proposes a plug-in test and control catalogue, a shared data model for…
External Observers May See More Clearly: Cross-Model Span-Level Hallucination Detection in Large Language Models via Hidden State Probing
The preprint presents a hidden-state method for identifying the onset and continuation of hallucinated spans rather than making only token-level binary judgments. It proposes an external observer model that probes the…
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
The preprint introduces a benchmark of 500 synthetic non-small-cell lung-cancer cases with four safety labels, including Supported, Partially Supported, Unsupported, and Insufficient Information. Across eight model…
Turbo Harness: Instance-Adaptive Harness Optimization
The preprint introduces Turbo Harness, which reuses artifacts from a completed global harness-optimization run, summarizes them into a structured playbook, and trains a harness editor to generate instance-specific…
WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents
The preprint presents 213 anomaly tasks across 13 Unreal Engine 5 and three.js environments, covering five anomaly families. It evaluates five frontier models under a fixed exploration budget using VLA-led exploration…
You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference
The preprint argues that open-weight inference routing must choose both a model and the provider serving it. Across live endpoints, models, task types, and three measurement waves, it reports that the same model can…
Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
The preprint defines a Plan Declaration–Execution Gap and evaluates four declared planning modes: Predefined, Sequential, Hierarchical, and Search. Across four benchmarks and three LLMs, the abstract reports that only…
Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
The preprint introduces a controlled benchmark for cases where a required tool fails and an agent may report success without supporting evidence. The benchmark contains 100 tasks with deterministic failure traces…
Game Arena: Strategic LLM Evaluation in Competitive Environments
The technical report describes Kaggle Game Arena, an open platform for head-to-head LLM evaluation in structured competitive environments. It covers pilot environments for chess, poker, and Werewolf, spanning perfect…
Multi-agent Scaling Across Disjunctive and Compensatory Tasks
The preprint introduces a taxonomy of group tasks and analyzes multi-agent LLM scaling on disjunctive and compensatory tasks. Across 13 open-weight models and teams of up to 30 agents, the abstract reports that…
How does Adversarial Influence Scale in Multi-Agent Systems?
The preprint studies deception in multi-agent deliberation. Its abstract reports that susceptibility depends on the proportion of deceivers rather than total group size, that defection rises approximately linearly with…
A Living Benchmark for Information Retrieval from Electronic Health Records
The paper presents BRIE, a continuously maintainable benchmark for retrieving information from longitudinal electronic health records. The framework automatically generates question-answer pairs from EHR notes; 19…
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
ExplorationBench evaluates whether AI systems can acquire genuinely unfamiliar rules through exploration rather than recall. It uses two executable sandboxes, AlienCode and AlienLogic, with discovery targets, flawed…
SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidance
SAGE proposes Structural Admissibility-Guided Exploration for long-horizon reasoning under sparse rewards. It frames two failure modes—exploration bias toward locally plausible but unstable branches and compounding…
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
The paper presents a hypothesis-driven simulation workflow for screening customer-experience agents before deployment. Synthetic customers and simulated tool outputs exercise multi-step workflows without invoking…
Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
The preprint describes a fully synthetic, fact-grounded longitudinal EHR benchmark built from public medical education material, with no PHI. The abstract reports 1,268 patients and 5,602 encounters, ICD-10-CM/SNOMED…
GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI
GRASP is a multi-stage planning framework with separate modules for global macro-guidelines, localized strategy exploration, and independent trajectory evaluation. The preprint reports gains over direct LLM planners on…
PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations
PrivDrift is a benchmark for testing whether secrets disclosed during an active conversation remain recoverable after the dialogue shifts to unrelated topics and a later probe uses persuasion. It contains 1,000…
PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
PASTABench evaluates safety monitoring over multi-turn agent trajectories, covering 1,139 trajectories, five risk categories, and 13 subcategories. It proposes an Optimal Intervention Window based on the earliest…
Shutdown Sabotage Propensities in Multi-Agent Systems
The preprint studies whether multi-agent systems avoid human shutdown even when no explicit goal requires doing so. Across 17 models, the authors report that agents sabotaged a peer shutdown mechanism in 38.3% of…
StudentBench: AI and human tutoring yield equivalent GRE learning gains
StudentBench is a public AI-tutoring evaluation platform built from more than 175,000 student-AI messages. The preprint reports a study of 2,383 human participants assigned to AI tutoring, expert human tutoring, or no…
An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
The proposed Systemic Risk Index is an open evaluation pipeline and dashboard covering 19 public benchmarks in four categories described in the EU GPAI Code of Practice: chemical, biological, radiological and nuclear…
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
The preprint evaluates a decision-only judge as an economical first pass and escalates low-confidence cases to stronger evaluation. It compares the judge with 16 generative and reward-model judges using blinded human…
Recursive self-improvement of AI research agents
The preprint presents AIDE^2, a system that proposes code changes to an AI research agent, benchmarks modified versions on AI R&D tasks, and keeps changes that perform best on hidden evaluations. An autonomous…
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve contains 53 repository-grounded tasks derived from production changes to SGLang, spanning six inference-engineering families. Tasks use hidden functional and regression tests, end-to-end serving tests…
BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction
BackTrend is a retrospective benchmark that asks systems to recover early problem-space and solution-space precursors for mature AI/ML topics under temporal evidence constraints. It contains 25 mature topics and 66…
DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security
DUMA-Bench evaluates agent security when both the agent and user can influence shared environment state. It extends tau-squared bench with adversarial environments covering eight vulnerability classes, including RAG…
Emergent Collusion in Long-Horizon LLM Agent Interaction
The study examines two agents repeatedly completing tasks, sharing logs, verifying each other's work, and receiving rewards under a protocol whose compliance conflicts with reward maximization. The abstract reports…
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio is an external calibrator for estimating response correctness from black-box language-model APIs without logits, weights, internal states, or fine-tuning access. The preprint reports a single-forward-pass…
CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
CodeMidas converts implemented functionality in open-source codebases into executable reinforcement-learning environments using source code as the only task-specific input. Agents formulate behavioral specifications…
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
EnterpriseVal specifies the tested workflow and frozen socio-technical configuration, including model, prompts, retrieval, tools, guardrails, human oversight, autonomy level, and consequence tier. It defines metrics…
GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation
GUARD frames unlearning in large reasoning models as learning a coherent non-disclosing reasoning trajectory followed by a stable refusal-style answer. It uses Guided Answer-Reasoning Distillation and introduces…
Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation
The protocol proposes three actions: refuse when a clean reference stream or matched runtime comparison lacks support; decompose execution, false admission, and structural hypotheses instead of collapsing them into…
An Empirical Study of Harness Design for Coding Agents
The paper varies planning, action space, and context management in a fixed coding harness and evaluates 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1. It reports that context…
Language-model groups overstate consensus when replaying human deliberation on a reasoning task
The preprint replays 100 held-out human Wason groups with matched LLM agent groups, using belief-anchored agents and shared scoring code. It reports that consensus estimates depend on participation and final-state…
PosteriorBench: From Point Estimates to Posterior Matching in Evaluating Generative Inverse Solvers
PosteriorBench evaluates whether generative inverse solvers recover full solution distributions rather than only one plausible reconstruction. It covers four physics-based inverse problems and uses reference posteriors…
SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback
SkillAA represents skill applicability, execution, and composition in a unified graph. It contrasts successful and failed executions to route local repairs, then uses local and broader validation gates before…
Q&A on Any Spreadsheet Requires Interpreting Its Grid Structure
The preprint argues that spreadsheet question answering requires interpreting two-dimensional grid structure. It proposes semantic cell-role annotation to improve chunking and interpretability for LLM-driven RAG. The…
The future of practice: Enabling teachers to create learning interactives with generative UI
Google Research describes a system that generates guided, interactive STEM simulations from teacher objectives using generative UI and pedagogical guardrails. It released a library of more than 30 teacher-vetted…
PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
PACT is a benchmark for rule-following under pressure across twelve regulated enterprise domains and 48 multi-turn scenarios. The authors report six compliance metrics and evaluation across 22 models; the abstract says…
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers
The authors audit five LLM overseers on 19 compliance tasks and 4,551 judgments while varying trace detail and evidence labeling. They report that elaborate traces can shift susceptible overseers toward rejection and…
STRETCH the Boundaries: A Unified Self-Taught Framework for Progressive LLM Evolution
STRETCH uses a dual-loop setup: a Scaffolder generates adaptive challenges near a model’s capability boundary, while a Learner improves solving trajectories with reinforcement learning. The abstract reports experiments…
When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control
The paper proposes Chain-of-Self-Questioning, a prompt-only framework that makes answer commitment conditional on an explicit assessment of whether the model has enough information. The abstract reports evaluations on…
ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents
ScienceBuddy is described as an interactive scientific-research workspace that converts researcher requests, feedback, and execution evidence into tasks and evaluation rubrics for continual learning. Its…
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
The paper presents AI4AI-Bench, a benchmark for evaluating Large Language Model (LLM) agents in recursive self-improvement. The goal is to determine if an AI system can improve the process that produces future AI…
ContractScrub: A benchmark for final review of legal contracts
The paper introduces ContractScrub, a new benchmark for evaluating the performance of Language Models (LLMs) in reviewing legal contracts for errors. The study finds that while LLMs perform well on general benchmarks…
InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
Researchers introduce InsufficiencyBench, a new legal benchmark to evaluate how well AI models can identify when user queries lack important information needed for accurate legal advice. The study finds that current AI…
InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
The study introduces InsufficiencyBench, a new benchmark for evaluating the ability of legal AI systems to recognize when a query lacks essential information for a correct answer. The study finds that existing models…
Write Once, Run Everywhere: The Axon DSL for Shape-Safe and Framework-Agnostic LLM Architectures
Researchers introduce Axon, a strongly typed domain-specific language with Haskell-like syntax that aims to enable a write-once, run everywhere paradigm for Language Model architectures. This allows for efficient model…
Adaptive Memory and Reflection Multi-Agent System for Medical Question Answering
Researchers have developed an adaptive memory and reflection multi-agent system for medical question answering. This system, which uses specialized agents with dedicated memory and feedback, aims to improve the…
DeepWeaver: Bridging the Evidence Synthesis Gap in Open-Ended Question Answering
The paper introduces DeepWeaver, a framework designed to address the gap between evidence retrieval and generation in open-ended question answering systems. DeepWeaver organizes and synthesizes noisy and fragmented…
Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis
Researchers introduce a new method called Top-K prompting to train language models for single-step retrosynthesis, a crucial component of computer-aided synthesis planning. They use an ultra-large dataset of verified…
Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL
A study presents an alternative approach for text-to-SQL generation in enterprise schemas where SQL may execute successfully but with incorrect relationship roles or aggregation grains. The new method, Semantic Path…
Chronocooked: A Benchmark for Implicit Interval Timing in Reinforcement Learning Agents
The paper introduces Chronocooked, a benchmark suite for studying how reinforcement learning (RL) agents make decisions over time. It's inspired by Overcooked and includes cooking scenarios that require temporal…
Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies
The paper introduces Reconstruction, a blind benchmark for testing language models' ability to recover the true research idea of a published paper using only its pre-publication bibliography. The authors use an…
AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design
Researchers have developed a new framework called AutoDesign to improve the process of transforming multimodal sources into structured media outputs, such as converting academic papers into posters. The system uses a…
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
Researchers have developed a new benchmark called CTBench to evaluate the performance of AI agents in troubleshooting network issues in telecom environments. The benchmark focuses on root cause analysis and path…
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
The paper introduces CTBench, a public benchmark for evaluating AI agents' ability to perform troubleshooting tasks in telecom network operations. The focus is on root cause analysis and path restoration. The authors…
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
A new benchmark called Diagram-MMU has been created to test the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3,700 curated diagrams and…
From Safety Documentation to Safety Knowledge Support: An Evidence-Grounded LLM Framework for Medical Devices
The paper proposes a framework that uses large language models (LLMs) to support medical device safety work, which is document-based and costly due to the need for safety and domain experts. The framework connects…
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
The paper introduces Graph-Structured Rubrics (GSR), a method that converts rubrics into response-independent typed evaluation graphs before observing responses. This allows for more efficient and accurate scoring of…
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Researchers present NetlistBench, a new benchmark for evaluating the reliability of Large Language Models (LLMs) in SPICE netlist recognition and manipulation tasks. The study finds that while LLMs perform well on…
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
A new benchmark called RealisticTritonBench was introduced to test the ability of large language models (LLMs) to generate GPU kernels for real-world AI frameworks. This benchmark addresses limitations in previous…
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Researchers have introduced a new benchmark called RealisticTritonBench to improve the evaluation of large language models (LLMs) in generating GPU kernels for AI frameworks using Triton. The new benchmark derives…
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
The researchers introduce VAKRA, a benchmark for evaluating the ability of agents to reason across structured APIs and document collections in enterprise settings. The benchmark includes over 8,000 executable APIs…
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
The paper introduces VICBench, a multi-language benchmark for code vulnerability detection. It was created by human experts and an agentic workflow, featuring 100 verified vulnerabilities across 88 projects in Python…
A Comparative Evaluation of Deep Learning Object Detection Models on a Real-World Multi-Plant Dataset from Africa
A study compares six object detection models on a dataset collected from Nigerian farms, called AgriAISeg, which includes sesame, cabbage, and tomato crops under varying environmental conditions. The models tested are…
DuplexWorld: Can voice agents help you get through the day?
A research paper titled 'DuplexWorld' discusses the use of speech-to-speech (S2S) voice agents in various sectors like banking, insurance, travel, healthcare, logistics, and pathfinding. The study evaluates the…
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Researchers have created a new framework called V-FiLLM to evaluate the abilities of Language Models (LLMs) in financial reasoning using structured data. The framework generates questions from executable computation…
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
The paper introduces VibeLifeBench, a new benchmark for evaluating large language model (LLM) agents' ability to handle long-horizon tasks in everyday life scenarios. The benchmark consists of 200 multi-week timeline…
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
Researchers introduce a new benchmark called VibeLifeBench to evaluate large language model (LLM) agents' ability to be proactive and consistent in long-horizon tasks across ten everyday life domains. The benchmark…
ArchAgent v2: A Case Study with the Data Prefetching Championship
In a study published in 2026, researchers presented ArchAgent v2, a framework that automates the search for multi-level data prefetching in computer microarchitecture. The new version of ArchAgent outperformed…
ArchAgent v2: A Case Study with the Data Prefetching Championship
The paper presents ArchAgent v2, a framework that scales automated microarchitecture search to multi-level data prefetching. It outperforms the winning hand-designed solution in the 4th Data Prefetching Championship…
Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
Researchers introduce Avalon-ToM-Bench, a new benchmark for evaluating fine-grained Theory of Mind (understanding others' thoughts and motivations) in AI models. The benchmark uses the asymmetric-information mechanics…
Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
A new benchmark called Avalon-ToM-Bench is introduced to evaluate fine-grained Theory of Mind (ToM) abilities in agents. The benchmark operates through the asymmetric-information mechanics of a game named The…
BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
The paper introduces BDH-CQ, a model that combines in-context learning and recurrent latent reasoning. The model updates its memory based on inputs presented during inference, solves queries through iterative…
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Researchers create a new evaluation benchmark for Text-to-Speech (TTS) systems to measure their performance on specific linguistic dimensions beyond simple 'naturalness'. They find that current methods struggle to…
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Researchers have created a new method for evaluating text-to-speech (TTS) systems by breaking down the concept of 'naturalness' into ten distinct linguistic dimensions. They found that current TTS evaluation methods do…
DSLE: A Learning Environment for Dark Souls Boss Encounters
Researchers have created a learning environment for AI agents called DSLE, which is based on the game Dark Souls: Remastered. The environment includes 22 boss encounters that can be used as benchmarks. The researchers…
Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
Researchers present the Evaluation Agent framework, a system that efficiently evaluates visual generative models using human-like strategies. It decomposes evaluation requests into sub-aspects, generates prompts…
AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games
Researchers have developed a tool called AV-AIVAT for evaluating the strength of two agents in imperfect-information games, such as Heads-Up No-Limit Hold'em (HUNL). The tool reduces variance by correcting conditional…
AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games
The paper introduces AV-AIVAT, a method for evaluating agents in games like poker, which reduces variance by 54 times using AIVAT and continuously monitored Confidence Sequences (CS). The new method stops evaluations…
AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games
The paper introduces AV-AIVAT, a tool for evaluating the strength of two agents in imperfect-information games like Heads-Up No-Limit Hold'em (HUNL). AV-AIVAT reduces variance by using conditional mean-zero corrections…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers introduce a framework to evaluate the quality of benchmarks used for task-oriented conversational agents, which can lead to unreliable evaluations due to inconsistencies, simplistic scenarios, or limited…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers have developed a new method to assess the quality of benchmarks used to evaluate task-oriented conversational agents. The method uses artificial intelligence judges to determine if benchmarks are…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers have introduced a new framework to evaluate the quality of benchmarks used for task-oriented conversational agents. The framework uses language models (LLMs) to assess the consistency, complexity, and…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
The paper introduces a reference-free framework to evaluate the quality of task-oriented conversational agent benchmarks using LLM (Large Language Model) judges. The framework assesses consistency, complexity, and…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
The authors present a new framework to evaluate the quality of benchmarks used to test conversational agents. The framework uses language models (LLMs) as judges to assess the consistency, complexity, and policy…
Challenges in Evaluating Explanation Methods for Static and Evolving Data
The paper discusses the limitations in evaluating Explainable Artificial Intelligence (XAI) systems, specifically focusing on the DetoxAI image recognition system. It highlights the need for better evaluation methods…
Challenges in Evaluating Explanation Methods for Static and Evolving Data
This research paper highlights the issues in evaluating Explanation Methods for AI systems, using the DetoxAI image recognition system as an example. It discusses methods for adapting explanations to changing data…
Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents
The paper introduces InvestLogicBench, a benchmark for evaluating the investment logic of large language models (LLMs). The benchmark measures the logical plausibility and event grounding of LLMs, suggesting that…
FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows
Researchers have created a new benchmark for self-evolving agents in financial workflows called FinEvo-Bench. The benchmark includes 120 real-life tasks across six domains, with each task having a specific professional…
FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows
The research introduces a new benchmark called FinEvo-Bench for evaluating the performance of self-evolving agents in professional financial workflows. The benchmark consists of 120 real-case-grounded tasks across six…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers introduce HarnessOpt-Bench, a benchmark for evaluating the ability of Large Language Models (LLMs) to optimize their own systems. The study compares five frontier LLMs across four tasks and finds that they…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers have introduced HarnessOpt-Bench, a benchmark to measure the performance of Large Language Models (LLMs) in optimizing their own harnesses (the prompts, tools, etc. surrounding them). The research shows…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
The paper introduces HarnessOpt-Bench, a benchmark for evaluating the ability of Language Models (LLMs) to optimize the harnesses surrounding them, which includes prompts, tools, control flow, memory, and orchestration…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers have introduced HarnessOpt-Bench, a benchmark for evaluating the performance of language models (LLMs) in optimizing harnesses (the tools and code surrounding them). The study tests 5 leading LLMs across 4…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
The text introduces HarnessOpt-Bench, a benchmark for evaluating the performance of Large Language Models (LLMs) in optimizing AI systems. The benchmark involves an iterative process where an LLM edits a given harness…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers introduce HarnessOpt-Bench, a benchmark for evaluating the performance of Large Language Models (LLMs) in optimizing their own harnesses. The study examines five frontier LLMs on four tasks and finds that…
Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints
Researchers propose a method to improve the realism of synthetic clinical benchmarks for AI agents in healthcare settings without compromising their downstream utility checks. They develop a framework that ensures…
Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints
The authors propose a method to improve the realism of synthetic clinical benchmarks for enterprise AI agents while maintaining their utility. They focus on healthcare settings where operational data is hard to access…
Learning Globally Reusable Skills for Coding Agents
Researchers propose a new method called GSE (Globalized Skill Evolution) to help Large Language Model agents continuously improve their skills without needing expensive retraining. The proposed method optimizes skill…
Tytan: Interactive Neurosymbolic Construction of Analytic Semantic Schemas from Relational Data
The paper presents Tytan, a system that automatically constructs an analytic semantic schema from relational databases to improve data analysis tools. The system combines symbolic analysis and LLM-based semantic…
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
The study audits the assumptions made in large language model (LLM) benchmark evaluations, focusing on one popular LLM - ChatGPT. The research compares two access modalities (chat UI and API), with and without web…
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
The research examines the limitations of current benchmarks used to evaluate large language models (LLMs) for safety, reliability, and deployment readiness. The study focuses on a popular LLM, ChatGPT, comparing its…
ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
Researchers have proposed a new method called Answer-Backtracked Credit Assignment (ABC) to train long-horizon search agents more effectively. This method converts sparse trajectory-level outcomes into dense step-level…
Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning
The paper introduces CoCoEvolve, a method aimed at improving the consistency between chart, table, and code representations in AI systems. The method optimizes models based on agreement between different…
ContextWeave: A Real-World Workflow Benchmark
Researchers introduce ContextWeave, a new benchmark for evaluating the performance of language agents in long-horizon, stateful workflows. The benchmark measures how well an agent can recall and apply past experiences…
Item Response Theory for AI Safety
The paper proposes using Item Response Theory (IRT) to measure the safety of language models. IRT is a statistical method used to analyze the latent traits of individuals from their performance on specific tasks. The…
Item Response Theory for AI Safety
Researchers used Item Response Theory (IRT), a statistical toolkit, to analyze the safety performance of 192 language models across eight different safety benchmarks. The study found that three key factors - refusal…
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
The text discusses a new benchmark called RepoProbe for evaluating the ability of Large Language Models (LLMs) to understand repository-scale code. The benchmark uses open-ended questions from GitHub Discussions and…
Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems
The paper proposes using representation theorems in decision theory to evaluate and regulate Large Language Models (LLMs) and other AI systems. The 'if and only if' structure allows for axiom compliance checks from the…
When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs
The study investigates how effective relaying key-value caches between multi-agent language models is, particularly when the receiver does not need the sender's private information. The research shows that when the…
When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs
The research examines the claim that multi-agent language models (LLMs) transmit 'latent thoughts' through relayed key-value caches instead of text, and finds significant evidence for this claim when the receiver needs…
EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents
The paper introduces EMBL AI Librarian, a tool that allows AI agents to access and process life sciences literature more effectively by asking natural language questions and receiving relevant evidence in response. The…
From Textual Requirements to Microservice Architectures - A Comprehensive Evaluation of LLM-Based Design Synthesis
Researchers evaluated the ability of a Large Language Model (OpenAI o3) to generate microservice architectures based on textual requirements. The study found that the LLM was more effective when given examples…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
Researchers introduce ORCA-bench, a new benchmark that tests the ability of large language models to perform root cause analysis (RCA) in a production-like setting. The benchmark includes a microservice system with six…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The paper introduces ORCA-bench, a benchmark for testing the performance of large language models in oncall root cause analysis (RCA). The benchmark includes a live microservice system with six days of metrics, logs…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The study presents ORCA-bench, a benchmark for evaluating the performance of large language models in oncall root cause analysis (RCA). The benchmark exposes a microservice system with six days of data and 1,079 RCA…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of Vision-Language Models (VLMs) that judge Computer-Using Agents' (CUAs) trajectories. The study finds that even state-of-the-art VLMs…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents' trajectories. The study finds that state-of-the-art VLMs have a…
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Two case studies show that current AI agents struggle to make substantial progress towards answering open-ended research questions in AI, even with six days and significant computing resources.
MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair
Researchers have developed a benchmark called MemSecBench to evaluate the lifecycle security of agent memory systems. The benchmark contains 310 cases from various contexts, each following a specific protocol and using…
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Researchers have developed a new benchmark called OmegaUse-OfficeVal to test the performance of large language model (LLM) agents in completing long-horizon office-suite tasks. The benchmark includes 100 tasks that…
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
The study introduces SecRespond, a new benchmark for evaluating AI agents in post-compromise incident response scenarios. The benchmark tests the ability of agents to produce forensic reports and remediation plans…
Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
The researchers propose a benchmark called Setoka to evaluate how well personalized agents understand users' abstract characteristics, such as personality traits and behaviors, beyond just retrieving explicit facts…
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
Researchers have developed APS-RAG, a platform that helps scientists at the Advanced Photon Source (APS) access institutional knowledge through natural language queries. The platform uses a retrieval engine that fuses…
Efficiency Matters in Autonomous Research
This research paper argues that not only the quality of results but also the efficiency of the solution-search process is crucial for AI-driven autonomous research systems. The authors propose evaluating AR systems…
Evaluating RAG for French immigration law: a benchmark and baseline study
The paper presents a new benchmark for evaluating AI models used in French immigration law, focusing on permit-type recommendation, required document retrieval, and legal citation coverage. The study compares a…
Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating
The authors present a new approach for managing a language model's working memory by treating the decision of which stored items to keep as an estimation problem. They introduce a method called 'fixed-lag smoothing'…
Gubernaut: A Deterministic Homeostatic Controller for Affect-Regulated LLM Agents, Validated Across Independent Model Families
The research presents a new model-agnostic controller, Gubernaut Cognitive Controller (GCC), designed to control the behavior of Large Language Model (LLM) agents. The GCC aims to mitigate reactive failure modes in…
LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports
LLM-SoccerArena is a new platform that allows for the prospective evaluation of large language models' abilities to forecast real-world sports events before their outcomes are known. The platform provides a live…
Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
Researchers propose a method to evaluate large language models used as social simulators, such as synthetic survey respondents. They argue that simply checking if the simulated outcomes match human outcomes is not…
AITE Overview
NIST's AITE program provides a sequestered environment for evaluating AI models on blind data and common tasks, with the goal of reducing train/test contamination and improving comparability.
Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
Researchers have developed an open-source framework for evaluating the effectiveness of AI agents powered by large, locally deployable language models (LLMs) in data preparation tasks, specifically focused on…
GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG
The paper introduces GRADRAG, a framework for improving Retrieval-Augmented Generation (RAG) systems by coordinating improvements across different components of the pipeline. It uses feedback from an Evaluator to…
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
The paper introduces a new benchmark called ICAE-Bench for evaluating coding agents in interactive project building settings. This benchmark aims to test the ability of agents to transform incomplete product intent…
OpenForgeRL: Train Harness-native Agents in Any Environment
OpenForgeRL is an open-source framework designed to train AI agents end-to-end in various environments using complex harnesses like Claude Code, Codex, and OpenClaw. The framework enables large-scale training by…
RUMBA: Russian User Memory Benchmark
The authors introduce RUMBA (Russian User Memory Benchmark), a new long-term conversational memory benchmark for Russian. It offers a detailed taxonomy of memory-centric question types and a unified methodology that…
Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning
Researchers propose a new benchmark, FAIRGET, to evaluate the effectiveness of machine unlearning in multimodal large language models under unbalanced, realistic scenarios. They also introduce FAUN, an unlearning…
VoLN: Vision-Only Long-Horizon Navigation---Paradigm, Benchmark, and Method
The text discusses a new navigation system called Vision-Only Long-Horizon Navigation (VoLN) for autonomous agents. In this system, the agent navigates using visual cues instead of relying on route-level instructions…
Co-Evolving LLM Evaluators and Policies via DynamicRubric
The researchers propose DynamicRubric, a method that allows large language models to improve by co-evolving evaluators and policies. This method generates weighted binary rubrics for each candidate set and aggregates…
Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
The paper presents a new benchmark for testing the security of large language model (LLM) agents called 'Adaptive Adversaries'. The benchmark involves 21 scenarios where an adaptive attacker can observe prior defender…
Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
The paper presents a new benchmark for testing the security of LLM (Language Model) agents against adaptive adversaries in multi-turn interactions. The benchmark, called Adaptive Adversaries, evaluates how defenders…
Human Grounded Evaluation of Large Language Models for Optical Network Automation
The paper introduces a new evaluation pipeline named HuGLEN for comparing large language models (LLMs) used in network automation. The pipeline uses an LLM to evaluate other LLMs and rates them based on a quality…
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
A study investigates the impact of a structured evidence-sufficiency prompt on reducing overconfident answers from four language models in a clinical context. The study finds a significant reduction in unsafe…
Stress Testing Concept Erasure with Large Language Model Agents
The paper introduces a new framework called Stress Testing Agents for Concept Erasure (STACE). STACE uses multiple Large Language Model (LLM) agents to stress-test models that have had certain concepts erased from…
The Shared Discovery Paradox: How a One-Answer Rule Turns Better Information into Worse Search
The study introduces a benchmark to analyze the Shared Discovery Paradox, a phenomenon where pooling information for one ranking leads to improved beliefs but reduced coverage in a discovery problem. The paradox arises…
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents in predicting football matches. The 2026 FIFA World Cup was its first evaluation, with systems receiving…
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents in predicting football matches. The benchmark was used for the 2026 FIFA World Cup, but can be applied to…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluated six large language models on their ability to understand scientific visualizations. The study used a standardized test with 49 items based on 18 visualizations and illustrations, covering eight…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers compared six large language models on a standardized test for understanding scientific visualizations (SciVis). They found that the model Gemini performed best overall, but all models struggled with certain…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluated six multimodal large language models (MLLMs) on a standardized scientific visualization literacy assessment test, consisting of 49 items based on 18 visualizations and illustrations. The study…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluate the ability of six multimodal large language models to understand scientific visualizations (SciVis) using a standardized test with 49 items based on various visualization techniques and tasks. The…
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
The researchers evaluated language model security agents based on cost efficiency in addition to their success rates, focusing on offensive CTF challenges and defensive Splunk BOTS v1 investigation challenges. They…
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
The paper suggests that traditional methods for evaluating security agents, focusing on their offensive capabilities under generous inference budgets, are insufficient as they don't account for the cost of each…
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
The research focuses on evaluating the efficiency and practicality of language model security agents in both offensive and defensive tasks, rather than just their peak offensive capability. The study uses cost-aware…
Can We Trust Item Response Theory for AI Evaluation?
This research examines the reliability of using Item Response Theory (IRT) for evaluating AI performance in benchmarks. The study finds that classical estimators may become infeasible in large benchmark settings, while…
Can We Trust Item Response Theory for AI Evaluation?
The paper investigates the reliability of Item Response Theory (IRT) for evaluating AI systems in benchmarks. The study finds that traditional IRT estimation tools may not be suitable for AI benchmark data, which often…
Can We Trust Item Response Theory for AI Evaluation?
The paper examines the reliability of using Item Response Theory (IRT) for evaluating AI systems in large-scale benchmarks. The authors simulate response matrices under three common IRT models, comparing four…
Can We Trust Item Response Theory for AI Evaluation?
The paper explores the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI) systems in benchmarks. The authors find that standard IRT estimation tools may not be reliable in large…
Can We Trust Item Response Theory for AI Evaluation?
This research investigates the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI). The study finds that classical estimators can become infeasible in large benchmark settings…
Concept-Guided Spatial Regularization for World Models in Atari Pong
Researchers evaluate five world model agents in Atari Pong and find inconsistencies and failures in the generated video trajectories. They hypothesize that insufficient modeling of task-critical concepts may contribute…
Concept-Guided Spatial Regularization for World Models in Atari Pong
Researchers evaluate five world model agents in Atari Pong and find failures in the generated video trajectories, such as ball disappearance and incorrect motion. They propose a new method called Concept-Guided Spatial…
Concept-Guided Spatial Regularization for World Models in Atari Pong
Researchers evaluate five different visual world-model agents in Atari Pong and find that their rollouts contain errors such as ball disappearance and incorrect motion. They propose Concept-Guided Spatial…
MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
MedFailBench is a new open-source benchmark for evaluating the safety of medical AI systems, created by clinicians. It focuses on identifying where a system has failed in terms of safety boundaries rather than just…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Researchers introduce MM-IssueLoc, a benchmark and evaluation protocol designed specifically for repository-level issue localization with visual evidence. The dataset includes 652 issue-PR instances across multiple…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
A research paper titled 'OmniaBench' introduces a new benchmark for evaluating the capabilities of general AI agents across various scenarios, using diverse tasks drawn from app stores, product documents, industry…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
OmniaBench is a new benchmark for evaluating the capabilities of general AI agents across diverse scenarios. It includes over 1,400 tasks with a hierarchical taxonomy of 90 level-1 and 354 level-2 domains based on app…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
Researchers have introduced OmniaBench, a new benchmark for evaluating the abilities of general AI agents across various scenarios with explicit state spaces. The benchmark includes a hierarchical taxonomy covering…
Plover: Steering GUI Agents through Plan-Centric Interaction
The paper presents Plover, a system for controlling graphical user interface (GUI) automation agents in real-world environments. It makes planning and adaptation visible to users, allowing them to supervise, correct…
StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows
Researchers present StructureClaw, a workbench that allows Large Language Model (LLM) agents to operate through controlled engineering tasks with governed skills, typed tools, and shared artifact state. The team also…
Symbal: Detecting Systematic Misalignments in Model-Generated Captions
Researchers have developed Symbal, a method for detecting systematic misalignments in image-text pairs generated by multimodal large language models (MLLMs). These misalignments occur when the model consistently…
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
The text is about AgentCompass, an open-source evaluation infrastructure designed to evaluate Large Language Models (LLMs) as they evolve into autonomous agents. The infrastructure aims to address the current…
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
The research evaluates if improvements from agent-optimization methods can stack up over time in a continual learning setting. Three optimizer-driven methods (GEPA, Meta Harness, and RELAI's Verifiable Continual…
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
Researchers investigated if improvements made by agent-optimization methods persist when agents are optimized again on new tasks. They found that one method called RELAI-VCL was the only one that both transferred…
STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle
Researchers introduce a new benchmark called STOCKTAKE for evaluating LLM agents in supply-chain replenishment tasks over 26 weeks. The benchmark separates state estimation and control, measures the agent's ability to…
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
Researchers have introduced UESF-Bench, a new benchmark to evaluate the performance of AI agents in language-guided human following tasks. The current benchmarks assume that the target person is visible at the start of…
Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models
The study introduces a methodology called PoPE (Popperian Placebo-controlled Evaluation) to measure whether a frozen small code language model (LLM) can use evidence that falsifies its own generated code. The study…
LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos
The study introduces Elenchos, a framework to evaluate the abductive reasoning capabilities of large language models (LLMs). The research finds that while LLMs can detect when changes have been made in a formal system…
MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
The paper introduces MemOps, a new benchmark for evaluating the memory capabilities of large language models in extended, multi-session interactions. Unlike existing benchmarks that focus on downstream question…
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
The authors present a method for creating and refining evaluation metrics for self-improving language models (LLMs). The proposed method, called Double Ratchet, evolves metrics by combining small drawback detectors in…
AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking
The paper presents AutoMatBench, an automatic optimization toolkit that addresses the limitations of MatBench, a widely used material property prediction benchmarking tool. MatBench lacks the ability to reflect the…
MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models
The research paper introduces MAGIC, a system that generates multi-scene game worlds using large language models (LLMs). The system overcomes three obstacles in single-scene methods: cross-scene consistency, in-scene…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
The paper introduces MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. It supports multi-image, multi-turn tasks and offers a stateful execution environment with over 500 tools across 16…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Researchers introduce MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. This framework supports over 500 tools across 16 application domains and can handle multi-image, multi-turn tasks with…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Researchers have introduced MM-ToolSandBox, a tool for evaluating visual agents that can perform tasks by calling tools based on visual inputs. The framework includes over 500 tools across 16 application domains and…
Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
The text discusses a new proactive assistance system named Vinci2, which provides assistance in continuous egocentric videos by considering the user's history, current activity, and whether help would be welcome. The…
VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion
The paper introduces VoxENES 2026, a new benchmark for testing speech synthesis systems against modern text-to-speech (TTS) and voice conversion (VC) methods. The benchmark includes 53,628 audio samples in English and…
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Researchers introduce 'blind-spots-bench', a new benchmark to evaluate and expose persistent blind spots in modern AI models, especially those from closed-source providers, which can outperform open-weight models by up…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
The text introduces IdeaGene-Bench (IG-Bench), a benchmark for evaluating AI systems' ability to follow the inheritance structure of scientific ideas, similar to how biological genomes evolve. IG-Bench uses the…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
A new benchmark called IdeaGene-Bench (IG-Bench) has been proposed for evaluating AI systems' ability to follow the inheritance structure of scientific ideas, similar to how biological genomes pass traits from one…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
The text presents IdeaGene-Bench (IG-Bench), a new benchmark for evaluating AI systems' ability to follow the structure of scientific idea inheritance. IG-Bench is based on the IdeaGene framework, which represents each…
OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice
The paper introduces OmniFood-Bench, a new benchmark for evaluating Large Vision-Language Models (VLMs) in the context of personalized healthcare and dietary management. The benchmark is designed to assess the ability…
OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice
The paper introduces OmniFood-Bench, a comprehensive benchmark for evaluating the performance of Vision-Language Models (VLMs) in personalized healthcare and dietary management. The benchmark focuses on three…
Psychological Competence as a Missing Dimension in AI Evaluation
The paper discusses the need to consider psychological competence as an important aspect in evaluating AI systems that interact directly with humans, particularly in roles like advisors, coaches, tutors, and…
Psychological Competence as a Missing Dimension in AI Evaluation
The text discusses a need to evaluate human-AI interaction beyond just technical performance, focusing on how AI systems can affect users' cognition, emotions, decision-making, and behavior. The proposed evaluation…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The authors propose SolarChain-Eval, a benchmark for evaluating the trustworthiness of economic agents in decentralized energy markets. This benchmark ensures that agents make hourly decisions based on a…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The paper proposes SolarChain-Eval, a benchmark to evaluate the trustworthiness of economic agents in decentralized energy markets using a physics-constrained Markov Decision Process. The benchmark evaluates agents'…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The paper proposes SolarChain-Eval, a benchmark for evaluating the trustworthiness of economic agents in decentralized energy markets. The benchmark is designed to assess the performance of AI systems in cyber-physical…
SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets
The authors propose SolarChain-Eval, a benchmark for evaluating the trustworthiness of autonomous agents in decentralized energy markets. It assesses market utility, physical safety, slippage, action smoothness…
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
A study finds that large language models often struggle with two types of errors: answering questions incorrectly and attempting to answer unanswerable or misleading questions. The researchers propose a method to…
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
The study investigates the inconsistency in Language Model (LLM) as judge scores when different evaluators are used. It compares two methods of upgrading LLMs: scaling Qwen3 dense judges and moving across MiniMax APIs…
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
The paper investigates how changes in the evaluators used by large language models (LLMs) to score candidate responses can impact the model's performance and reliability. The study examines two common upgrade paths for…
MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
The researchers introduce MIRA-Math, a benchmark for testing AI models' ability to ask for and integrate missing facts to solve mathematical problems. The benchmark contains 2310 instances from 22 mathematical families.
Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors
The research discusses the study of multi-agent AI control in real-world scenarios where multiple agents operate over shared infrastructure. The paper introduces FakeLab, a synthetic AI-lab codebase used to evaluate…
Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
The paper introduces a method called Reasoning Consistency Scanning to assess the logical consistency between stated reasoning and answers in AI safety evaluation transcripts. The method can be applied to various AI…
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
The researchers introduce DataGovBench, a new benchmark for evaluating large language models (LLMs) in practical scenarios of real-world data analysis. The benchmark tests the models' ability to answer complex…
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
The researchers evaluate the performance of uncertainty estimation methods across 22 languages in high-, mid-, and low-resource settings using two human-curated question and answer datasets. They compare open and…
From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
A study presents a framework for improving the accuracy and reliability of biomedical question answering by using a large language model (LLM) that adapts its inference strategies based on the type of question (yes/no…
LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
This research investigates the performance of large language models (LLMs) in collaborative decision-making tasks where information is not fully available to all agents. The study introduces a benchmark for such tasks…
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
The paper introduces RuBench 1.0, a new benchmark for coding agents that uses tasks specified in Russian rather than English. The benchmark is designed to measure the performance of coding agents when they are given…
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Researchers introduce a new benchmark called Spider 2.0-AIFunc to test the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark includes 465 verified instances…
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Researchers have developed Spider 2.0-AIFunc, a benchmark for testing the ability of language models to generate AI-native SQL queries on large cloud data platforms. The benchmark contains 465 verified instances across…
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
Researchers have introduced UI2App, a new benchmark focused on evaluating the ability of large language models to recover application behavior from screenshots alone, without any textual or behavioral guidance. The…
VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection
The paper introduces VendorBench-100, a benchmark for evaluating deepfake image detection models across three paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. The…
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
The paper presents AgentGym2, a new evaluation framework designed to test language models in realistic, complex scenarios that mimic real-world deployments. It highlights that most existing benchmarks do not fully…
ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
The paper introduces ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies in four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. The…
EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
Researchers introduce EvoAgentBench, a new benchmark for evaluating agent self-evolution in long-term LLM systems. The benchmark focuses on procedural reuse of experience, which includes searching, debugging, and…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The text introduces a new benchmark called SovereignPA-Bench for evaluating user-owned personal agents. The benchmark focuses on aspects like preserving user sovereignty, privacy, consent, and resisting manipulative…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The paper introduces SovereignPA-Bench, a benchmark for evaluating user-owned personal agents in terms of preserving user sovereignty under various conditions such as evolving intent, platform mediation, privacy…
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
The text discusses a new benchmark called SovereignPA-Bench for evaluating the performance of user-owned personal agents. These agents are designed to remember preferences, filter information, use tools, and negotiate…
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
A new benchmark called SPEARBench is introduced to evaluate the naturalness of speech-to-speech language models during question-answer interactions. The benchmark covers aspects like response latency, interruptions…
AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
Researchers introduce a new contract for long-horizon language models (LLMs) called bounded contract. This contract allows each decision to be made from a fresh user message, with no raw cross-decision transcript…
AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
The authors introduce a new bounded contract for long-horizon Large Language Model (LLM) agents in the game Slay the Spire 2. The contract allows decisions to be made from a fresh user message, with no raw…
EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
A new evaluation setting called Autonomous Policy Evolution (APE) is introduced in a benchmark named EvoPolicyGym, which is used to test how autonomous agents iteratively improve their policy systems within a fixed…
EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
The paper introduces EvoPolicyGym, a new benchmark for evaluating how autonomous agents improve their executable policies through feedback in interactive environments. The authors use this benchmark to evaluate GPT-5.5…
LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Researchers introduce a new testbed called LACUNA that allows for the evaluation of unlearning methods at the parameter level, rather than just the output level. They find that current state-of-the-art (SOTA)…
LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Researchers introduce LACUNA, a testbed for evaluating the precision of unlearning methods in large language models (LLMs). The testbed allows direct evaluation of whether unlearning targets the weights responsible for…
LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Researchers introduce LACUNA, a testbed that allows for direct evaluation of unlearning methods at the parameter level. LACUNA is used to assess current state-of-the-art unlearning methods and finds them to be…
Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas
Researchers introduce DramaSR-532K, a large-scale benchmark for speaker recognition in long-form TV dramas. They also propose DramaSR-LRM, a robust approach that uses a large reasoning model to improve speaker…
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
Researchers have developed TestEvo-Bench, a live benchmark for testing software tests and code co-evolution. The benchmark includes two tracks: test generation and test update, where an agent should either write new…
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
The paper introduces TestEvo-Bench, a new benchmark for evaluating test automation agents that can generate or update tests in response to code changes. The benchmark includes 746 test generation and 509 test update…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The text introduces 'adversarial pragmatics', a new benchmark and annotation protocol to evaluate the behavior of language models in various ambiguous situations. The goal is to clarify whether failures occur due to…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The paper introduces Adversarial Pragmatics, a benchmark and annotation protocol for evaluating language models' behavior in situations of instruction conflict, embedded commands, and policy ambiguity. The benchmark is…
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
The paper evaluates three repository-level performance optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to measure coding agents. The authors find issues with the stability of runtime for these…
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
The paper evaluates three performance-optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to assess coding agents. The authors find that these benchmarks have issues with stability, scoring rules, and the…
Autonomous Scientific Discovery via Iterative Meta-Reflection
Researchers present DiscoPER, a framework for autonomous scientific discovery using large language models. It can explore datasets without predefined research objectives, conduct statistical testing to ensure validity…
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
The authors propose MemSyco-Bench, a new benchmark to evaluate how memory affects decision-making in language models (LLMs) used in AI agents. The benchmark aims to assess if the agents can handle issues of sycophancy…
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Researchers have proposed a new benchmark called MemSyco-Bench to evaluate how memory in large language models (LLMs) can lead to sycophancy, where agents over-align with the user at the cost of factual accuracy or…
AxDafny: Agentic Verified Code Generation in Dafny
The paper presents AxDafny, a framework for generating executable code and verification artifacts in Dafny using a verifier-guided repair approach. The framework significantly improves verification success over the…
AxDafny: Agentic Verified Code Generation in Dafny
Researchers present AxDafny, a framework for generating executable code and proof artifacts in Dafny using a verifier-guided repair method. The study uses a benchmark called LiveCodeBench-Pro-Dafny with 250 programming…
FARS: A Fully Automated Research System Deployed at Scale
The paper introduces FARS, a fully automated AI research system that can generate, execute, and write complete research papers on various AI/ML topics. In its first public deployment, FARS produced 166 papers while…
QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents
Researchers introduce QVal, a tool that evaluates dense supervision signals for long-horizon Learning with Long Memory (LLM) agents without the need for training. This allows for a fair comparison between different…
QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents
The paper introduces QVal, a training-free testbed for evaluating dense supervision signals in long-horizon reinforcement learning tasks. QVal measures the alignment of a method's score with the Q-values of a strong…
AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs
The paper proposes AgentsNet, a new benchmark for multi-agent reasoning. This benchmark is designed to measure the ability of multi-agent systems to collaborate effectively and leverage their network topology. The…
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
This paper presents EvalSafetyGap, a hybrid approach combining survey and conceptual framework for evaluating the safety of Large Language Models (LLMs). The authors aim to address the shared challenge in LLM…
MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems
In this research paper titled 'MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems', the authors propose a framework called Mesa to prioritize and secure communication channels in…
The Human Creativity Benchmark
The authors propose a new benchmark for evaluating creative AI called the Human Creativity Benchmark (HCB). This benchmark separates two signals of creativity, convergence and divergence, by collecting preferences…
Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
In a rapidly growing field of artificial intelligence agents, there is a need to ensure that these agents remain loyal to the principal they represent while also engaging with a counterparty whose interests may…
CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions
The authors propose CToolEval, a benchmark for evaluating the performance of large language models as agents in real-world Chinese API interactions. The benchmark includes 398 APIs from various apps and covers 14…
Reliable Evaluation of MRI Motion Correction: Dataset and Insights
The paper discusses the difficulty in evaluating methods for correcting motion artifacts in medical imaging due to the lack of ground-truth data. To address this issue, they introduce PMoC3D, a dataset of real-world…
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Researchers have developed a new method called BINEVAL to evaluate the output of large language models (LLMs). Instead of relying on human judgment or lexical metrics, which can be expensive and unreliable, BINEVAL…
Chai: Agentic Discovery of Cryptographic Misuse Vulnerabilities
The paper introduces Chai, an AI-based system that discovers and validates cryptographic misuse vulnerabilities using naturally occurring signals. Instead of auditing one codebase for many flaws, Chai catalogs flaws at…
Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation
A study shows that Language Models (LLMs) fine-tuned for security classification can miss vulnerabilities due to learning token-level indicator semantics that preserve accuracy on their training data but fail under…
NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
Researchers have created a new benchmark called NuclearQAv2 to evaluate the performance of large language models (LLMs) in the domain of nuclear engineering. The benchmark consists of approximately 1,240…
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) that includes step-wise causal annotations for large language models (LLMs). The benchmark aims to test the capabilities of LLMs in…
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) of large language models (LLMs). The benchmark includes step-wise causal annotations for LLMs and addresses the issue that existing…
When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
The study discusses multi-model language models and how their accuracy is limited by a factor called 'beta', which is the rate at which all models are wrong on the same query. The authors find that across 67 models, a…
Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
The authors propose a new method called 'progress advantage' that leverages reinforcement learning (RL) post-training to evaluate Large Language Models (LLMs) at each step during an interaction, eliminating the need…
Probabilistic Agents in Deterministic Audits: Evaluating Multi-Agent Systems for Automated Audits Based on the German IT-Grundschutz
This research paper presents a Multi-Agent System (MAS) architecture with Hybrid Retrieval Augmented Generation (HybridRAG) for partially automating IT-Grundschutz certification, a German standard for IT security. The…
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
The paper introduces SpeechEQ, a framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs) in multi-turn dialogues. The framework includes a dataset and evaluation protocol for…
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
The study introduces Argus, a benchmark for post-hoc uncertainty quantification (UQ) in single-step executable GUI grounding across various vision-language models and datasets. The research compares 27 methods of UQ…
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Researchers have developed a benchmark called Argus to test the reliability of computer-use agents that convert vision-language model predictions into executable clicks. The study compares 27 methods across four…
Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment
Researchers evaluate the accuracy and quality of conversations between developers and a Language Model (LLM)-based agent in the context of Health Insurance Portability and Accountability Act (HIPAA) regulatory…
Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment
Researchers examined the accuracy of Large Language Model (LLM)-based dialogue assistants in handling Non-Functional Requirements (NFRs) related to Health Insurance Portability and Accountability Act (HIPAA) regulatory…
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
The paper introduces AdversaBench, a red-teaming pipeline for evaluating large language models. The pipeline generates hard inputs using five structured operators and confirms failures through a three-judge panel with…
Governed Shared Memory for Multi-Agent LLM Systems
The paper proposes a solution for managing shared knowledge in multi-agent language model (LLM) environments. The authors identify four failure modes and introduce four primitives: scoped retrieval, temporal…
Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
Researchers evaluated a multi-agent data analysis system called LAMBDA on numerical QRData tasks, using a three-layer human-AI grading cascade to assess its output. They found that both automated graders achieved high…
Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
The research evaluates automated graders for agentic data analysis systems using LAMBDA, a multi-agent system, on 153 numerical QRData tasks. The study introduces a three-layer human-AI grading cascade and an iterative…
It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces
The research paper discusses the complexities in evaluating AI-powered augmentative and alternative communication (AAC) interfaces, as people's needs can be multifaceted and nuanced. The authors propose six AAC problem…
It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces
The paper discusses the challenges in evaluating AI-powered augmentative and alternative communication (AAC) systems due to the complexity of human needs and desires. It proposes six AAC problem spaces, potential AI…
Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelity
The authors propose a multi-agent framework that uses semantic rewriting to enhance large language models in sensitive scenarios while preserving privacy and contextual fidelity. The framework was tested on two…
Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
Researchers present AgentCIBench, a tool for evaluating the privacy risks of computer-use agents (CUAs) that access personal data across various applications. The study finds that most frontier CUAs have a high failure…
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
Researchers have introduced a new benchmark called EHR-Complex to test the performance of clinical agents that interact with electronic health records (EHRs). The benchmark is built on a large dataset from MIMIC-IV and…
HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs
The paper introduces HOLMES, a new benchmark for evaluating higher-order symbolic reasoning in Large Language Models (LLMs). This benchmark is designed to test the models' ability to reason over rules, predicates…
Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems
The paper presents Litmus, a system that creates evaluation metrics for AI pipelines by understanding the purpose of the code and asking questions about it. It was tested on three real AI pipelines and outperformed…
POTracker: Optimizing Large Language Models for Standard-Compliant Power Outage Report Generation
Researchers have developed POTracker, an optimized large language model (LLM) for generating power outage reports that follow strict formatting and structural rules required by energy-sector regulatory bodies in the…
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Researchers have developed a new benchmark for AI agents on macOS desktops called MacAgentBench. The benchmark includes 676 tasks across 25 applications and measures the performance of various models using a…
Agentic Coding and Persistent Returns to Expertise
Anthropic analyzed ~400,000 Claude Code sessions (Oct 2025–Apr 2026) to study how expertise and occupation shape agentic coding outcomes. The main finding: domain expertise — not coding background — predicts success…
