Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
3 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on ai benchmarking
3 notes
How Benchmarks Mis-Score Computer-Use Agents
The paper discusses issues with the benchmark scores of Computer-Use Agents (CUA), highlighting that the scores are often inaccurate due to brittle scripted oracles. The authors propose a reliability framework and…
MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
The paper introduces MemOps, a new benchmark for evaluating the memory capabilities of large language models in extended, multi-session interactions. Unlike existing benchmarks that focus on downstream question…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
Researchers have developed IdeaGene-Bench (IG-Bench), a benchmark for assessing AI systems' ability to follow the inheritance structure of scientific ideas, similar to how biological genomes pass traits from generation…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
