ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang
• Venue: arXiv: cs.CV
• Published: 2026-07-27
• Collected: 2026-07-28
• URL: http://arxiv.org/abs/2607.24743v1
• PDF: https://arxiv.org/pdf/2607.24743v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have developed a new multimodal large language model called ClinFusion for understanding medical information from various image sources. The model outperforms leading open-source models and proprietary ones in several benchmarks, including visual question answering, report generation, and instruction following. A group of board-certified radiologists found that ClinFusion's generated reports were the highest ranked, and a new evaluation metric proposed by the researchers showed strong correlation with expert judgment.

What the Document Says

• The paper introduces ClinFusion, a vision-centric MLLM for holistic medical understanding
• ClinFusion uses a Cascade Spatial-Aware Locality Fusion operator in its compositional and cascaded vision encoder architecture
• The researchers propose MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation
• ClinFusion sets a new state-of-the-art across multiple 2D and 3D multimodal medical benchmarks
• The model outperforms leading open-source models such as Hulu-Med, Lingshu on 20 out of 24 benchmarks
• ClinFusion demonstrates better multimodal capabilities than powerful proprietary models like GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks
• The model can be augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows
• A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports
• The RoI-grounded metric proposed by the researchers shows strong correlation with expert judgment among all automatic evaluation metrics

Key Concepts Mentioned

• ClinFusion
• multimodal large language model
• vision-centric
• holistic medical understanding
• Cascade Spatial-Aware Locality Fusion operator
• MedIF-Bench
• region-of-interest-grounded method
• evaluation framework

Claims Made by the Document

• ClinFusion sets a new state-of-the-art in multimodal medical benchmarks
• ClinFusion outperforms leading open-source models on 20 out of 24 benchmarks
• ClinFusion demonstrates better multimodal capabilities than powerful proprietary models on 13 out of 16 benchmarks
• A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports
• The RoI-grounded metric proposed by the researchers shows strong correlation with expert judgment

Evidence Used

• outperforming leading open-source medical MLLMs ( extit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks
• demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks
• A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports
• validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined

Methods / Approach

• introducing a compositional and cascaded vision encoder architecture
• proposing a Cascade Spatial-Aware Locality Fusion operator
• introducing MedIF-Bench for instruction-following assessment
• introducing a region-of-interest-grounded method

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The limitations of the authors are not explicitly stated in the source.

Open Questions

• What specific improvements does ClinFusion bring to the multimodal medical benchmarks compared to other models?
• How does ClinFusion perform on unmentioned or more recent benchmarks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper discusses ClinFusion's performance on a set of 2D and 3D multimodal medical benchmarks, but it is not clear how the model compares to others in terms of specific improvements or its performance on unmentioned or more recent benchmarks.

Raw Abstract / Excerpt

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.