Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

Source

arXiv

arxiv.org/abs/2607.27084v1

Read the source

Source

Plain-Language Brief

Researchers have created SciFigQual-Bench, a new benchmark for evaluating the quality of scientific figures in computer science papers. The benchmark assesses five dimensions: clarity, layout, caption fit, context relevance, and misleading risk. It includes data from top computer science conferences from 2020 to 2025, with a total of 6,308 images scored by multiple domain experts. A new evaluation framework, SFQ-Agent, has been designed to enable automated scoring on this benchmark. The GPT-5.6-Sol model used in SFQ-Agent performed best among several tested models.

What the Document Says

  • Researchers have created a new benchmark called SciFigQual-Bench for evaluating the quality of scientific figures in computer science papers.
  • The benchmark assesses five dimensions: clarity, layout, caption fit, context relevance, and misleading risk.
  • The data covers top computer science conferences from 2020 to 2025, with a total of 6,308 images scored by multiple domain experts.
  • SFQ-Agent is an evaluation framework designed for automated scoring on the SciFigQual-Bench benchmark.
  • GPT-5.6-Sol model used in SFQ-Agent performed best among several tested models.

Key Concepts Mentioned

  • Scientific images
  • IQA methods
  • SciFigQual-Bench
  • computer-science conferences
  • dimensions
  • data
  • SFQ-Agent
  • GPT-5.6-Sol

Claims Made by the Document

  • Existing IQA methods are not suitable for scientific papers.
  • Previous studies on scholarly charts have limitations in verification of caption alignment, citation relevance, or visual misleadingness.
  • SciFigQual-Bench is a new benchmark that addresses these limitations.
  • SFQ-Agent achieves the lowest overall average absolute error and highest consistency rate on the test subset eval1200.
  • GPT-5.6-Sol model used in SFQ-Agent outperformed other models.

Evidence Used

  • Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers.
  • Existing IQA methods are predominantly designed for natural photographs or AI-generated content.
  • The few existing studies on scholarly charts remain confined to visual-surface comparisons.
  • We propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions.
  • SFQ-Agent is designed to achieve auditable and refined scoring through the collection and fusion of modal evidence.
  • Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%).

Methods / Approach

  • Design of SciFigQual-Bench benchmark
  • Creation of SFQ-Agent evaluation framework

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The performance of the GPT-5.6-Sol model may not generalize to other domains or types of scientific papers.

Open Questions

  • How does SciFigQual-Bench compare with existing benchmarks for IQA in natural photographs or AI-generated content?
  • What are the potential applications of SciFigQual-Bench beyond computer science?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper focuses on scientific figures in computer science papers. It is unclear how well the proposed methods would apply to other fields or types of figures.

Raw Abstract / Excerpt

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.