evaluation benchmark
5 notes, newest first.
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
The paper discusses Diagram-MMU, a new benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3.7k diagrams and…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The paper introduces OSReward, a benchmark for evaluating the reliability of vision-language models (VLMs) used as judges for computer-using agents (CUAs). The benchmark consists of CUA trajectories from various agent…
SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context
Researchers have created SciFigQual-Bench, a new benchmark for evaluating the quality of scientific figures in computer science papers. The benchmark assesses five dimensions: clarity, layout, caption fit, context…
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
The paper introduces ICAE-Bench, a benchmark for evaluating coding agents in interactive project building settings. Instead of being limited to completing code under fully specified instructions, these agents are…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Researchers have introduced a new benchmark called MM-IssueLoc for evaluating the effectiveness of multimodal repository-level issue localization, which includes visual evidence such as screenshots, error dialogs, and…
