ai benchmark
6 notes, newest first.
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
AI4AI-Bench is a new benchmark for assessing the ability of AI agents to design their own training algorithms. The benchmark consists of 10 frozen research repositories covering 10 different algorithm families. In each…
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
The study introduces VAKRA, a benchmark for evaluating the ability of agents to reason across APIs and document collections in enterprise settings. The benchmark includes over 8000 executable APIs across 62 domains…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers introduce HarnessOpt-Bench, a benchmark for evaluating the ability of Large Language Models (LLMs) to optimize their own systems. The study compares five frontier LLMs across four tasks and finds that they…
ContextWeave: A Real-World Workflow Benchmark
The paper introduces ContextWeave, a new benchmark for evaluating the performance of language agents in long-horizon, stateful workflows. The benchmark focuses on memory as an essential component and measures workspace…
InfoOps Bench: A live information operations safety benchmark
The paper introduces a dynamic AI benchmark that measures the integrity of language models against being used for state-backed information operations. The benchmark uses data from over 2,100 information operations from…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
A research paper titled 'OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios' introduces a new benchmark called OmniaBench, designed to evaluate the capabilities of general AI agents in diverse…
