Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
43 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on benchmarking
43 notes, page 1 of 2
Write Once, Run Everywhere: The Axon DSL for Shape-Safe and Framework-Agnostic LLM Architectures
Researchers introduce Axon, a strongly typed domain-specific language with Haskell-like syntax that aims to enable a write-once, run everywhere paradigm for Language Model architectures. This allows for efficient model…
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
A new benchmark called Diagram-MMU has been created to test the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3,700 curated diagrams and…
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Researchers have introduced a new benchmark called RealisticTritonBench to improve the evaluation of large language models (LLMs) in generating GPU kernels for AI frameworks using Triton. The new benchmark derives…
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
The paper introduces VibeLifeBench, a new benchmark for evaluating large language model (LLM) agents' ability to handle long-horizon tasks in everyday life scenarios. The benchmark consists of 200 multi-week timeline…
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
Researchers introduce a new benchmark called VibeLifeBench to evaluate large language model (LLM) agents' ability to be proactive and consistent in long-horizon tasks across ten everyday life domains. The benchmark…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers have introduced HarnessOpt-Bench, a benchmark to measure the performance of Large Language Models (LLMs) in optimizing their own harnesses (the prompts, tools, etc. surrounding them). The research shows…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers have introduced HarnessOpt-Bench, a benchmark for evaluating the performance of language models (LLMs) in optimizing harnesses (the tools and code surrounding them). The study tests 5 leading LLMs across 4…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
The text introduces HarnessOpt-Bench, a benchmark for evaluating the performance of Large Language Models (LLMs) in optimizing AI systems. The benchmark involves an iterative process where an LLM edits a given harness…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Researchers introduce HarnessOpt-Bench, a benchmark for evaluating the performance of Large Language Models (LLMs) in optimizing their own harnesses. The study examines five frontier LLMs on four tasks and finds that…
Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints
Researchers propose a method to improve the realism of synthetic clinical benchmarks for AI agents in healthcare settings without compromising their downstream utility checks. They develop a framework that ensures…
Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints
The authors propose a method to improve the realism of synthetic clinical benchmarks for enterprise AI agents while maintaining their utility. They focus on healthcare settings where operational data is hard to access…
ContextWeave: A Real-World Workflow Benchmark
Researchers introduce ContextWeave, a new benchmark for evaluating the performance of language agents in long-horizon, stateful workflows. The benchmark measures how well an agent can recall and apply past experiences…
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
The text discusses a new benchmark called RepoProbe for evaluating the ability of Large Language Models (LLMs) to understand repository-scale code. The benchmark uses open-ended questions from GitHub Discussions and…
Evaluating RAG for French immigration law: a benchmark and baseline study
The paper presents a new benchmark for evaluating AI models used in French immigration law, focusing on permit-type recommendation, required document retrieval, and legal citation coverage. The study compares a…
LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports
LLM-SoccerArena is a new platform that allows for the prospective evaluation of large language models' abilities to forecast real-world sports events before their outcomes are known. The platform provides a live…
AITE Overview
NIST's AITE program provides a sequestered environment for evaluating AI models on blind data and common tasks, with the goal of reducing train/test contamination and improving comparability.
Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
Researchers have developed an open-source framework for evaluating the effectiveness of AI agents powered by large, locally deployable language models (LLMs) in data preparation tasks, specifically focused on…
Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning
Researchers propose a new benchmark, FAIRGET, to evaluate the effectiveness of machine unlearning in multimodal large language models under unbalanced, realistic scenarios. They also introduce FAUN, an unlearning…
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents in predicting football matches. The 2026 FIFA World Cup was its first evaluation, with systems receiving…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers compared six large language models on a standardized test for understanding scientific visualizations (SciVis). They found that the model Gemini performed best overall, but all models struggled with certain…
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
The research focuses on evaluating the efficiency and practicality of language model security agents in both offensive and defensive tasks, rather than just their peak offensive capability. The study uses cost-aware…
Can We Trust Item Response Theory for AI Evaluation?
This research examines the reliability of using Item Response Theory (IRT) for evaluating AI performance in benchmarks. The study finds that classical estimators may become infeasible in large benchmark settings, while…
Can We Trust Item Response Theory for AI Evaluation?
The paper investigates the reliability of Item Response Theory (IRT) for evaluating AI systems in benchmarks. The study finds that traditional IRT estimation tools may not be suitable for AI benchmark data, which often…
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
A research paper titled 'OmniaBench' introduces a new benchmark for evaluating the capabilities of general AI agents across various scenarios, using diverse tasks drawn from app stores, product documents, industry…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
