Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

Source

arXiv

arxiv.org/abs/2608.12262v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Weihao Bo, Shan Zhang, Yanpeng Sun, Jie Liu, Yongke Yao, Jinhao Du, Wei He, Kai Zou, Zechao Li, Jingdong Wang
  • Venue: arXiv: cs.CV
  • Published: 2026-08-12
  • Collected: 2026-08-13
  • URL: http://arxiv.org/abs/2608.12262v1
  • PDF: https://arxiv.org/pdf/2608.12262v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

A new benchmark called Diagram-MMU has been created to test the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3,700 curated diagrams and 18,300 human-validated questions across six domains. It evaluates MLLMs on three tasks: diagram-to-code parsing, diagram-to-code editing, and diagram question answering. The evaluation of 12 MLLMs shows that diagram-to-code tasks are more challenging than diagram question answering, and Claude-4.6 Opus consistently improves across all three tasks under agentic settings.

What the Document Says

  • Diagram-MMU is a new benchmark for assessing MLLMs' ability in scientific diagram parsing and understanding.
  • The benchmark includes 3,700 curated diagrams and 18,300 human-validated questions across six domains.
  • It evaluates MLLMs on three tasks: diagram-to-code parsing, diagram-to-code editing, and diagram question answering.
  • The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering.
  • Models struggle to parse and edit diagrams well, indicating a need for methods to enhance MLLMs' capability in diagram-to-code generation.
  • Under agentic settings, most models improve parsing and editing performance but degrade on question answering.
  • Claude-4.6 Opus consistently improves across all three tasks.

Key Concepts Mentioned

  • Diagram-MMU
  • MLLMs
  • scientific diagrams
  • LaTeX TikZ code
  • benchmark
  • evaluation
  • diagram-to-code parsing
  • diagram-to-code editing
  • diagram question answering
  • agentic settings
  • Claude-4.6 Opus

Claims Made by the Document

  • Diagram-MMU is a new benchmark for MLLMs' ability in scientific diagram parsing and understanding.
  • The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering.
  • Models struggle to parse and edit diagrams well.

Evidence Used

  • Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains.
  • The evaluation of 12 MLLMs is presented in the paper.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.