Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Source

arXiv

arxiv.org/abs/2608.06270v1

Read the source

Source

Plain-Language Brief

The study examines the 'thinking-with-images' paradigm used in multimodal Language Models (LLMs), which includes visual operations like crop-and-zoom. The research finds that these operations often result in minimal or negative improvements over direct inference, and can even fail on certain questions. To understand if visual evidence returned by these operations affects the answer, the authors create a causal graph to separate observation-mediated paths from action-induced shortcuts. They find two failure modes: 'Calling Without Looking' and 'Looking Without Planning'. The former indicates that visual evidence has no effect on the answer, while the latter shows that observations are informative but the call schedule is inconsistent. The study also identifies a discrepancy they call the 'illusion of visual tool-use', where aggregate accuracy gains are observed, but visual tool-use is not causally effective across many rollouts.

What the Document Says

  • The 'thinking-with-images' paradigm is used in multimodal LLMs with active visual operations like crop-and-zoom.
  • Models using these operations often achieve minimal or negative gains over direct inference and can fail on questions.
  • The study investigates if returned visual evidence causally affects the answer by formulating visual tool-use as a causal graph.
  • Two failure modes are identified: 'Calling Without Looking' and 'Looking Without Planning'.
  • A discrepancy called the 'illusion of visual tool-use' is found, where aggregate accuracy gains exist but visual tool-use is not causally effective across many rollouts.

Key Concepts Mentioned

  • thinking-with-images
  • multimodal LLMs
  • crop-and-zoom
  • visual operations
  • causal graph
  • policy
  • trajectory
  • step
  • Visual Evidence Gain

Claims Made by the Document

  • Models using visual operations often achieve minimal or negative gains over direct inference.
  • Visual evidence returned by these operations does not always affect the answer.
  • Two failure modes, 'Calling Without Looking' and 'Looking Without Planning', exist in visual tool-use.

Evidence Used

  • The authors state that models using visual operations often achieve only marginal or negative gains over direct inference at substantially higher token cost.
  • They mention that models may repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The 'thinking-with-images' paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.