Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

Source

arXiv

arxiv.org/abs/2607.27084v1

Read the source

Source

Plain-Language Brief

Researchers propose SciFigQual-Bench, a benchmark for evaluating the quality of scientific figures in computer science papers across five dimensions. The dataset includes over 6,000 images from top conferences between 2020 and 2025, with each image associated with its caption, citing sentence, and manuscript context. A new evaluation framework, SFQ-Agent, is also introduced to enable automated assessment of the benchmark using large models like GPT-5.6-Sol.

What the Document Says

  • Scientific images are analyzed in this research.
  • Existing IQA methods are not suitable for scientific papers.
  • Previous studies on scholarly charts only focus on visual-surface comparisons.
  • SciFigQual-Bench is proposed as a full-text contextual benchmark for evaluating scientific images across five dimensions: clarity, layout, caption fit, context relevance, and misleading risk.
  • The data covers top computer-science conferences from 2020 to 2025 with 6,308 images scored by multiple domain experts.
  • Gold-standard annotations are created through the aggregation of expert scores.
  • SFQ-Agent is designed as a staged cross-modal evaluation framework for automated evaluation on the benchmark.
  • GPT-5.6-Sol is used in SFQ-Agent for achieving auditable and refined scoring.
  • The performance of multiple mainstream large models is evaluated on a test subset eval1200.
  • SFQ-Agent (F3) with GPT-5.6-Sol outperforms other evaluation schemes with an overall average absolute error of 0.418 and a consistency rate of 93.4%.

Key Concepts Mentioned

  • Scientific images
  • Image quality assessment (IQA)
  • SciFigQual-Bench
  • Full-text contextual benchmark
  • Computer science conferences
  • Gold-standard annotations
  • SFQ-Agent
  • Staged cross-modal evaluation framework
  • GPT-5.6-Sol
  • Test subset eval1200

Claims Made by the Document

  • Existing IQA methods are not suitable for scientific papers.
  • SciFigQual-Bench addresses the limitations of previous studies on scholarly charts.
  • SFQ-Agent achieves better performance than other evaluation schemes.

Evidence Used

  • The text discusses that existing IQA methods are not suitable for scientific papers and previous studies on scholarly charts have limitations.
  • The introduction of SciFigQual-Bench, its five dimensions, and the data it covers is detailed in the text.
  • The performance of SFQ-Agent is compared to other evaluation schemes.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.