Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
8 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on benchmarks
8 notes
A Living Benchmark for Information Retrieval from Electronic Health Records
The paper presents BRIE, a continuously maintainable benchmark for retrieving information from longitudinal electronic health records. The framework automatically generates question-answer pairs from EHR notes; 19…
An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
The proposed Systemic Risk Index is an open evaluation pipeline and dashboard covering 19 public benchmarks in four categories described in the EU GPAI Code of Practice: chemical, biological, radiological and nuclear…
Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies
The paper introduces Reconstruction, a blind benchmark for testing language models' ability to recover the true research idea of a published paper using only its pre-publication bibliography. The authors use an…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers introduce a framework to evaluate the quality of benchmarks used for task-oriented conversational agents, which can lead to unreliable evaluations due to inconsistencies, simplistic scenarios, or limited…
OpenForgeRL: Train Harness-native Agents in Any Environment
OpenForgeRL is an open-source framework designed to train AI agents end-to-end in various environments using complex harnesses like Claude Code, Codex, and OpenClaw. The framework enables large-scale training by…
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
Researchers have introduced UESF-Bench, a new benchmark to evaluate the performance of AI agents in language-guided human following tasks. The current benchmarks assume that the target person is visible at the start of…
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
The researchers introduce DataGovBench, a new benchmark for evaluating large language models (LLMs) in practical scenarios of real-world data analysis. The benchmark tests the models' ability to answer complex…
Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
In a rapidly growing field of artificial intelligence agents, there is a need to ensure that these agents remain loyal to the principal they represent while also engaging with a counterparty whose interests may…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
