FIELD NOTES TOPIC
ai benchmarking
3 notes, newest first.
How Benchmarks Mis-Score Computer-Use Agents
The paper discusses issues with the benchmark scores of Computer-Use Agents (CUA), highlighting that the scores are often inaccurate due to brittle scripted oracles. The authors propose a reliability framework and…
MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
The paper introduces MemOps, a new benchmark for evaluating the memory capabilities of large language models in extended, multi-session interactions. Unlike existing benchmarks that focus on downstream question…
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
Researchers have developed IdeaGene-Bench (IG-Bench), a benchmark for assessing AI systems' ability to follow the inheritance structure of scientific ideas, similar to how biological genomes pass traits from generation…
