ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Source

Source name: arXiv
Source kind: paper
Authors / organization: Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang
Venue: arXiv: cs.CV
Published: 2026-07-27
Collected: 2026-07-28
URL: http://arxiv.org/abs/2607.24743v1
PDF: https://arxiv.org/pdf/2607.24743v1
DOI: UNKNOWN

Plain-Language Brief

The paper introduces ClinFusion, a system designed for holistic medical understanding using multimodal large language models (MLLMs). ClinFusion addresses challenges in the medical domain by unifying diverse 2D and native 3D medical image understanding. It also proposes a vision-grounded evaluation framework and outperforms other open-source medical MLLMs on various benchmarks, including visual question answering, report generation, and instruction following.

What the Document Says

The paper presents ClinFusion, a system for holistic medical understanding using multimodal large language models (MLLMs).
ClinFusion addresses limitations in the medical domain by proposing a compositional and cascaded vision encoder architecture.
This architecture includes a Cascade Spatial-Aware Locality Fusion operator for unifying diverse 2D and native 3D medical image understanding within a fused encoder.
The paper also introduces a vision-grounded evaluation framework.
This includes MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation.
ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks.
It outperforms leading open-source medical MLLMs such as Hulu-Med and Lingshu on 20 out of 24 benchmarks.
ClinFusion demonstrates multimodal capabilities better than powerful proprietary models like GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks.
The system can be augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows.
A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports.
The RoI-grounded metric is validated as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.

Key Concepts Mentioned

ClinFusion
Multimodal large language models (MLLMs)
Vision-centric
Holistic medical understanding
Evaluation framework
MedIF-Bench
Region-of-interest-grounded method
Board-certified radiologists

Claims Made by the Document

ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks.
ClinFusion outperforms leading open-source medical MLLMs on 20 out of 24 benchmarks.
ClinFusion demonstrates multimodal capabilities better than powerful proprietary models on 13 out of 16 benchmarks.
The RoI-grounded metric achieves the strongest correlation with expert judgment among all automatic evaluation metrics examined.

Evidence Used

ClinFusion outperforms other open-source medical MLLMs and proprietary models on various benchmarks as stated in the text.

Methods / Approach

Compositional and cascaded vision encoder architecture
Cascade Spatial-Aware Locality Fusion operator
Vision-grounded evaluation framework
MedIF-Bench
Region-of-interest-grounded method

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The text does not provide information about the limitations of the agent (ClinFusion) as described by the authors.

Open Questions

What are the specific details of the compositional and cascaded vision encoder architecture?
What is the Cascade Spatial-Aware Locality Fusion operator exactly?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper provides a high-level overview of ClinFusion but does not delve into specific details about its architecture.

Raw Abstract / Excerpt

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.