Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
5 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on evaluation benchmark
5 notes
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
The paper discusses Diagram-MMU, a new benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3.7k diagrams and…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The paper introduces OSReward, a benchmark for evaluating the reliability of vision-language models (VLMs) used as judges for computer-using agents (CUAs). The benchmark consists of CUA trajectories from various agent…
SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context
Researchers have created SciFigQual-Bench, a new benchmark for evaluating the quality of scientific figures in computer science papers. The benchmark assesses five dimensions: clarity, layout, caption fit, context…
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
The paper introduces ICAE-Bench, a benchmark for evaluating coding agents in interactive project building settings. Instead of being limited to completing code under fully specified instructions, these agents are…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Researchers have introduced a new benchmark called MM-IssueLoc for evaluating the effectiveness of multimodal repository-level issue localization, which includes visual evidence such as screenshots, error dialogs, and…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
