The work of visual grounding sits in under 2% of attention heads — masking the top 20 drops grounding accuracy by up to 80 points

1Which part of the model is doing the work

A vision-language model finds the part of an image a sentence points to and reflects that in its answer. The behavior itself is easy to observe; what happens inside the model to produce it was not understood. This work takes inspiration from retrieval heads, a finding on the language-model side, and asks whether vision-language models hold something analogous. The answer is yes, and the authors name them Visual Retrieval Heads.

2The numbers behind the localization

The control is what makes the claim work. That performance collapses when the top 20 are masked could, on its own, just be the effect of removing 20 heads worth of capacity. Only the comparison — that masking the same number of random heads barely matters — establishes that those 20 are special. The design is built to support a causal claim.

3Organizing the method of finding them

A second contribution is tidying up how heads get scored. Existing methods are recast under a unified design space — which query tokens to look from, how to aggregate keys, how to aggregate across samples — and within that space the combination that most reliably surfaces causal heads is identified: scoring attention from output prediction tokens, summed over the ground-truth referent region.

Rather than lining methods up side by side, the work first puts them on common ground and then compares.

4How far the text-side findings carry

The authors report that the causal-sparse-universal triad established for text retrieval heads is replicated here, along with properties not previously reported. Although discovered through bounding-box prediction, the heads remain causal on attribute, spatial, counting and visual-math benchmarks. They are functionally specific, preserving output format while corrupting localization.

And they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector and instruction tuning. Of the 1,900 arXiv papers this site holds as of 2026-09-02, cs.CV is the largest primary category at 482. Where most of that work proposes new methods or performance, this one dissects the interior of existing models. This article is our own summary and does not warrant the correctness of the claims.