Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

Source

arXiv

arxiv.org/abs/2607.11818v1

Read the source

Source

Plain-Language Brief

Researchers introduce MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. This framework supports over 500 tools across 16 application domains and can handle multi-image, multi-turn tasks with realistic conversational phenomena. The study evaluates 12 state-of-the-art models from various systems and finds that they lack robust visual tool-calling capability, with even the best model achieving below a 50% success rate. The analysis shows that visual precision is a primary bottleneck for capable models, as approximately half of the failures are due to incorrect information extraction from images despite correct task workflows.

What the Document Says

  • The MM-ToolSandBox framework was introduced for evaluating visually grounded tool-calling agents.
  • The framework supports 500+ tools across 16 application domains and multi-image, multi-turn tasks.
  • It handles realistic conversational phenomena like goal revisions, error corrections, and state mutations.
  • An automated scenario generation pipeline produces diverse scenarios for interactive UI applications.
  • The study evaluates 12 state-of-the-art models from various systems.
  • Even the best model achieves below a 50% success rate in visual tool-calling tasks.
  • Failure analysis reveals that incorrect information extraction from images is a primary bottleneck for capable models, accounting for about 53% of failures.
  • The framework and benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox

Key Concepts Mentioned

  • MM-ToolSandBox
  • visually grounded tool-calling agents
  • evaluation framework
  • stateful execution environment
  • multi-image, multi-turn tasks
  • goal revisions
  • error corrections
  • state mutations
  • automated scenario generation pipeline
  • information-flow-guided planning
  • multi-stage quality filtering
  • human-verified nominal scenarios
  • interactive UI applications
  • state-of-the-art models
  • planning
  • precision
  • failure analysis
  • publicly available

Claims Made by the Document

  • Current models lack robust visual tool-calling capability.
  • Visual precision is a primary bottleneck for capable models.

Evidence Used

  • Even the best model achieves below a 50% success rate.
  • Failure analysis reveals that visual precision is a primary bottleneck, accounting for about 53% of failures.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox