Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
56 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on benchmark
56 notes, page 3 of 3
AxDafny: Agentic Verified Code Generation in Dafny
Researchers present AxDafny, a framework for generating executable code and proof artifacts in Dafny using a verifier-guided repair method. The study uses a benchmark called LiveCodeBench-Pro-Dafny with 250 programming…
AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs
The paper proposes AgentsNet, a new benchmark for multi-agent reasoning. This benchmark is designed to measure the ability of multi-agent systems to collaborate effectively and leverage their network topology. The…
The Human Creativity Benchmark
The authors propose a new benchmark for evaluating creative AI called the Human Creativity Benchmark (HCB). This benchmark separates two signals of creativity, convergence and divergence, by collecting preferences…
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) that includes step-wise causal annotations for large language models (LLMs). The benchmark aims to test the capabilities of LLMs in…
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) of large language models (LLMs). The benchmark includes step-wise causal annotations for LLMs and addresses the issue that existing…
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Researchers have developed a benchmark called Argus to test the reliability of computer-use agents that convert vision-language model predictions into executable clicks. The study compares 27 methods across four…
HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs
The paper introduces HOLMES, a new benchmark for evaluating higher-order symbolic reasoning in Large Language Models (LLMs). This benchmark is designed to test the models' ability to reason over rules, predicates…
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Researchers have developed a new benchmark for AI agents on macOS desktops called MacAgentBench. The benchmark includes 676 tasks across 25 applications and measures the performance of various models using a…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
