MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

Source

Source name: arXiv
Source kind: paper
Authors / organization: Kaixin Ma, Di Feng, Alexander Metz, Jiarui Lu, Eshan Verma, Afshin Dehghan
Venue: arXiv: cs.CV
Published: 2026-07-13
Collected: 2026-07-14
URL: http://arxiv.org/abs/2607.11818v1
PDF: https://arxiv.org/pdf/2607.11818v1
DOI: UNKNOWN

Plain-Language Brief

Researchers have introduced MM-ToolSandBox, a tool for evaluating visual agents that can perform tasks by calling tools based on visual inputs. The framework includes over 500 tools across 16 application domains and supports multi-image, multi-turn tasks. It also accounts for conversational phenomena such as goal revisions and error corrections. The study found that current state-of-the-art models are not robust in performing visual tool-calling tasks, with the best model achieving below a 50% success rate. The main issue is found to be the inaccurate extraction of information from images rather than planning issues.

What the Document Says

Introduction of MM-ToolSandBox: a framework for evaluating visually grounded tool-calling agents.
Framework provides stateful execution environment with over 500 tools across 16 application domains.
Supports multi-image, multi-turn tasks and handles conversational phenomena like goal revisions, error corrections, state mutations.
Automated scenario generation pipeline produces diverse scenarios for interactive UI applications.
Evaluation of 12 state-of-the-art models shows current models lack robust visual tool-calling capability, even the best achieving below 50% success rate.
Failure analysis reveals visual precision is a primary bottleneck: 53% of failures due to incorrect image information extraction despite correct task workflows.
A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, larger models fail at perceiving what they see.
The framework and benchmark are publicly available.

Key Concepts Mentioned

MM-ToolSandBox
visually grounded tool-calling agents
stateful execution environment
multi-image, multi-turn tasks
goal revisions
error corrections
state mutations
scenario generation pipeline
interactive UI applications
state-of-the-art models
success rate
visual precision
planning issues

Claims Made by the Document

MM-ToolSandBox is a useful framework for evaluating visually grounded tool-calling agents.
Current state-of-the-art models lack robust visual tool-calling capability.
Inaccurate extraction of information from images is the main issue in current models.

Evidence Used

Introduction and description of MM-ToolSandBox in text.
Evaluation results of state-of-the-art models.
Failure analysis results.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox