Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

Source

arXiv

arxiv.org/abs/2608.12262v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Weihao Bo, Shan Zhang, Yanpeng Sun, Jie Liu, Yongke Yao, Jinhao Du, Wei He, Kai Zou, Zechao Li, Jingdong Wang
  • Venue: arXiv: cs.CV
  • Published: 2026-08-12
  • Collected: 2026-08-13
  • URL: http://arxiv.org/abs/2608.12262v1
  • PDF: https://arxiv.org/pdf/2608.12262v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The paper discusses Diagram-MMU, a new benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3.7k diagrams and 18.3k human-validated questions across six domains. It assesses MLLMs on three tasks: diagram-to-code parsing, diagram-to-code editing, and diagram question answering under various settings. The evaluation shows that diagram-to-code tasks are more challenging and models struggle with parsing and editing diagrams, suggesting a need for methods to enhance MLLMs' capabilities in this area.

What the Document Says

  • Diagram-MMU is a benchmark for assessing Multimodal Large Language Models (MLLMs) in scientific diagram parsing and understanding.
  • It consists of 3.7k curated diagrams and 18.3k human-validated questions across six domains.
  • The benchmark evaluates MLLMs on three tasks: diagram-to-code parsing, diagram-to-code editing, and diagram question answering.
  • Models struggle more with diagram-to-code tasks compared to diagram question answering.
  • Most models improve in diagram parsing and editing under agentic settings but degrade on question answering.
  • Claude-4.6 Opus consistently improves across all three tasks.

Key Concepts Mentioned

  • Multimodal Large Language Models
  • Scientific Diagrams
  • Diagram-MMU
  • Benchmark
  • Parsing
  • Understanding
  • Three Tasks

Claims Made by the Document

  • Diagram-MMU is designed to assess MLLMs' ability for scientific diagram parsing and understanding.
  • Models struggle more with diagram-to-code tasks compared to diagram question answering.
  • Most models improve in diagram parsing and editing under agentic settings but degrade on question answering.
  • Claude-4.6 Opus consistently improves across all three tasks.

Evidence Used

  • The paper describes Diagram-MMU as a benchmark for assessing MLLMs' ability.
  • The paper states that models struggle with diagram-to-code tasks.
  • The evaluation results show most models improving in parsing and editing but degrading on question answering.
  • The conclusion of the paper indicates Claude-4.6 Opus consistently improves across all three tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.