Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Source

arXiv

arxiv.org/abs/2608.06270v1

Read the source

Source

Plain-Language Brief

Researchers audit the effectiveness of using visual tools like crop-and-zoom in multimodal language models by creating a causal graph to separate observation-mediated paths from action-induced shortcuts. They find that two failure modes exist: 'Calling Without Looking' and 'Looking Without Planning'. The first indicates that returned observations have no effect on the answer, while the second shows that observations are informative but the call schedule is incoherent.

What the Document Says

  • The study investigates the efficiency of visual tools in multimodal language models using active operations like crop-and-zoom.
  • Two failure modes, 'Calling Without Looking' and 'Looking Without Planning', were discovered where returned observations have no causal effect on the answer or are informative but the call schedule is incoherent.
  • The researchers formulated visual tool-use as a causal graph to separate observation-mediated paths from action-induced shortcuts and conducted interventions at three levels: policy, trajectory, and step.
  • They introduced Visual Evidence Gain, an estimand that isolates the contribution of each returned observation.

Key Concepts Mentioned

  • thinking-with-images paradigm
  • crop-and-zoom
  • multimodal LLMs
  • visual tool-use
  • causal graph
  • policy
  • trajectory
  • step
  • Visual Evidence Gain

Claims Made by the Document

  • Visual tool-use often results in marginal or negative gains over direct inference.
  • Models may repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly.
  • Policy miscalibration occurs with two failure modes: 'Calling Without Looking' and 'Looking Without Planning'.
  • Visual tool-use is not causally effective across a broad range of rollouts.

Evidence Used

  • The text describes the results from conducting interventions on six representative models and five fine-grained perception benchmarks.
  • The study uncovers the discrepancies between aggregate accuracy gains and the lack of causal effectiveness across various rollouts.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The 'thinking-with-images' paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.