Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence

Source

arXiv

arxiv.org/abs/2607.27066v1

Read the source

Source

Plain-Language Brief

Researchers have introduced SciFigAlign, a system for scoring scientific figures based on their ability to support the claims made in the manuscript they're part of. The system uses a dataset of over 3,000 figures from peer-reviewed conference papers, each rated along four dimensions: Clarity, Relevance, Informativeness, and Structure. SciFigAlign fine-tunes CLIP and SciBERT models to score figures by considering the figure crop, caption, citing paragraphs, and light paper context. The system significantly outperforms a baseline using large language models as judges, achieving a 59% relative error reduction.

What the Document Says

  • The study presents SciFigAlign, a system that scores scientific figures based on their alignment with the manuscript they're part of.
  • A dataset of 3,857 figures is used, each rated along four dimensions: Clarity, Relevance, Informativeness, and Structure.
  • SciFigAlign fine-tunes CLIP and SciBERT models using per-modality cross-attention and CubeMLP fusion to score figures based on figure crop, caption, citing paragraphs, and light paper context.
  • The system achieves a macro MAE of 0.3524 and within-paper pairwise accuracy of 81.64% when tested with n = 396, showing a 59% relative error reduction over the best LLM-as-judge baseline.
  • Ablations confirm that manuscript-grounded inputs, citing-context denoising, and ranking supervision are essential for scientific figure assessment.

Key Concepts Mentioned

  • Scientific figures
  • SciFigAlign
  • CLIP
  • SciBERT
  • Manuscript context
  • Figure quality assessment

Claims Made by the Document

  • SciFigAlign outperforms LLM-as-judge baselines for scientific figure scoring.

Evidence Used

  • The paper reports achieving a 59% relative error reduction over the best LLM-as-judge baseline.
  • Comparison results are provided against the LLM-as-judge baseline.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

  • What are the specific details of the dataset used in this study?
  • How does SciFigAlign compare to other existing methods for scientific figure assessment?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It is not explicitly stated how the dataset was created or what it includes.
  • Comparisons are made with an LLM-as-judge baseline but it's unclear if other methods were considered.

Raw Abstract / Excerpt

Scientific figure assessment in peer review differs fundamentally from general image quality evaluation: a figure must be visually legible, faithfully support the manuscript's claims, and communicate evidence with a clear visual hierarchy. However, if we apply traditional image assessment methods to scientific figure quality assessment, limitations emerge: classic IQA models capture perceptual quality or aesthetics but cannot judge whether a figure serves the paper's scientific argument; CLIP-based methods assess generic image-text correspondence, yet lack understanding of manuscript context; and zero-shot LLM/VLM judges, when repurposed for figure scoring, often yield overly concentrated scores with limited fusion of visual and textual evidence. We introduce an annotated dataset of 3,857 scientific figures from peer-reviewed conference papers, each rated along four peer-review-oriented dimensions: Clarity, Relevance, Informativeness, and Structure. We propose SciFigAlign, a fine-tuned multimodal scorer that grounds figure quality assessment in manuscript evidence. Given a figure crop, caption, citing paragraphs, and light paper context, SciFigAlign fine-tunes CLIP and SciBERT end-to-end with per-modality cross-attention and CubeMLP fusion, jointly optimizing SmoothL1 regression with a within-paper ranking hinge loss. Under paper-level splits, SciFigAlign achieves a macro MAE of 0.3524 and a within-paper pairwise accuracy of 81.64% on the test set with n = 396, a 59% relative error reduction over the best LLM-as-judge baseline with MAE 0.864. Ablations confirm that manuscript-grounded inputs, citing-context denoising, and ranking supervision are all critical, showing that scientific figure assessment requires learned alignment between visual content and manuscript evidence rather than prompting alone, even with state-of-the-art VLMs.