Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

Source

Source name: arXiv
Source kind: paper
Authors / organization: Yoav Baron, Sara Dorfman, Roni Paiss, Daniel Cohen-Or, Or Patashnik
Venue: arXiv: cs.CV
Published: 2026-07-07
Collected: 2026-07-08
URL: http://arxiv.org/abs/2607.06445v1
PDF: https://arxiv.org/pdf/2607.06445v1
DOI: UNKNOWN

Plain-Language Brief

Researchers investigate why Vision-Language Models (VLMs) struggle to maintain localization accuracy during image editing, even though they have strong localization capabilities on their own. The study suggests that this gap is due to the VLM's role as a condition encoder in editing pipelines, which restricts its autoregressive generation process. The authors introduce Analysis-by-Proxy, a framework that trains a lightweight proxy model using intermediate representations of the VLM for an auxiliary localization task. This allows them to analyze the VLM and discover where the localization information is encoded. The findings reveal a mismatch between how spatial knowledge is represented in the VLM condition encoder and how it is extracted by current editing pipelines.

What the Document Says

VLMs are used as conditioning backbone for diffusion-based image editing due to their multimodal reasoning capabilities.
Standalone VLMs have strong localization capabilities, but editing pipelines struggle to maintain this accuracy, particularly in complex scenes.
The authors hypothesize that the performance gap is because VLM is treated as a condition encoder which restricts its autoregressive generation process.
Analysis-by-Proxy framework is introduced to analyze the VLM through a lightweight, interpretable proxy model trained on VLM's intermediate representations for an auxiliary localization task.
The findings expose a mismatch between how spatial knowledge is represented in a VLM condition encoder and how it is extracted by current editing pipelines.
Under single-pass constraints, the localization signal does not reliably propagate to predefined layer configurations commonly used for conditioning.
Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt.

Key Concepts Mentioned

Vision-Language Models (VLMs)
diffusion-based image editing
localization capabilities
condition encoder
Analysis-by-Proxy
lightweight proxy model
spatial knowledge

Claims Made by the Document

VLMs struggle to maintain localization accuracy during image editing.
The performance gap is due to VLM being treated as a condition encoder.
Localization signal does not reliably propagate to predefined layer configurations under single-pass constraints.
Crucial localization signal remains hidden within intermediate representations.

Evidence Used

VLMs are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy...

Methods / Approach

Introduction of Analysis-by-Proxy framework
Training a lightweight proxy model on the VLM's intermediate representations for an auxiliary localization task

Limitations Stated by the Authors

The study does not provide specific details about the training data or the architecture of the proxy model used in the Analysis-by-Proxy framework.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The research is based on a single study and may not fully represent the entire field. Further studies could investigate other factors affecting the performance of VLMs during image editing.

Open Questions

What specific factors contribute to the mismatch between how spatial knowledge is represented in a VLM condition encoder and how it is extracted by current editing pipelines?
How can this mismatch be addressed to improve the localization accuracy of VLMs during image editing?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text does not provide specific details about the method used for training the proxy model or the evaluation metrics used for measuring localization accuracy.

Raw Abstract / Excerpt

Vision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy, particularly in complex, multi-entity scenes. In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder. In this role, the model is restricted to a single forward pass, preventing the autoregressive generation process for which it was optimized, thereby failing to fully expose its capabilities. To investigate whether this spatial understanding persists when the VLM is used as a condition encoder, we introduce Analysis-by-Proxy. In this framework, we train a lightweight, interpretable proxy model on the VLM's intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, we uncover the specific VLM representations that encode localization information. Our findings expose a fundamental mismatch between how spatial knowledge is represented within a VLM condition encoder and how it is extracted by current editing pipelines. We reveal that under single-pass constraints, the localization signal does not reliably propagate to the predefined layer configurations commonly used for conditioning. Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt. Using our introduced Analysis-by-Proxy framework, we reveal the fundamental failures of existing condition extraction strategies in editing pipelines, opening the door to more principled design of conditioning architectures.