Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy

Source

arXiv

arxiv.org/abs/2607.15176v1

Read the source

Source

Plain-Language Brief

Researchers evaluated six large language models on their ability to understand scientific visualizations. The study used a standardized test with 49 items based on 18 visualizations and illustrations, covering eight techniques and 11 task types. Results showed that the model called Gemini performed best overall, but the open-source models fell below human performance. Performance varied across techniques and tasks, with models excelling in scientific illustration, search, and spatial understanding, but struggling with texture-based and integration-based visualizations, and quantitative estimation.

What the Document Says

  • The study evaluates six multimodal large language models (MLLMs) on their understanding of scientific visualization (SciVis).
  • The evaluation is based on a standardized SciVis literacy assessment test with 49 items and 18 visualizations/illustrations.
  • The test covers eight techniques and 11 task types.
  • The study compares the performance of three closed-source models and three open-source models under a closed-world protocol.
  • Performance data from 485 human participants is used for comparison.
  • Gemini, one of the models, exceeded the human mean across the evaluated subsets.
  • The open-source models performed below the human baseline.
  • Performance varies significantly across techniques and tasks.
  • Models perform best on scientific illustration, search, and spatial understanding.
  • They struggle with texture-based and integration-based visualizations and quantitative estimation.
  • Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation.
  • The findings suggest that SciVis literacy is a necessary benchmark for evaluating multimodal AI systems.

Key Concepts Mentioned

  • Multimodal large language models
  • Scientific visualization literacy
  • Evaluation
  • Closed-source models
  • Open-source models
  • Human participants
  • Gemini

Claims Made by the Document

  • Current MLLMs do not exhibit uniform SciVis literacy.
  • Gemini is the strongest model overall.
  • Performance is highly uneven across techniques and tasks.

Evidence Used

  • Results from the study

Methods / Approach

  • The researchers used a standardized SciVis literacy assessment test

Limitations Stated by the Authors

The authors do not mention any limitations in their work.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent does not have enough context to identify potential limitations. For example, the study may be limited by the specific models chosen for evaluation or the specific aspects of SciVis assessed.

Open Questions

  • What are the specific reasons for the recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation?
  • How can the performance of open-source models be improved to reach human baseline?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study does not provide specific reasons for the recurring failures or suggestions for improving open-source model performance.

Raw Abstract / Excerpt

Multimodal large language models (MLLMs) are increasingly used to interpret visualizations, yet current evaluations remain largely chart-centric and provide limited evidence of understanding of scientific visualization (SciVis). We benchmark six MLLMs on the scientific visualization literacy assessment test, a standardized SciVis literacy assessment comprising 49 items based on 18 scientific visualizations and illustrations, spanning 8 techniques and 11 task types. We evaluate three closed-source and three open-source models under a closed-world protocol and compare their performance using data from 485 human participants. Results show that current MLLMs do not exhibit uniform SciVis literacy. Gemini is the strongest model overall, exceeding the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline. Performance is highly uneven across techniques and tasks: models perform best on scientific illustration, search, and spatial understanding, but struggle on texture-based and integration-based visualizations and on quantitative estimation. Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation. These findings position SciVis literacy as a necessary benchmark dimension for evaluating multimodal AI systems. Our code and model outputs are publicly available at https://github.com/patdmp/mllm-scivis-lit-benchmark.