2026
Narnaware, Vishal; Gupta, Animesh; Zhai, Kevin; Wang, Zhenyi; Shah, Mubarak
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs Conference
European Conference on Computer Vision (ECCV), Springer, 2026.
Abstract | Tags: ECCV, Hallucination Mitigation, Multimodal LLMs | Links:
@conference{narnaware2026visageb,
title = {Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs},
author = {Vishal Narnaware and Animesh Gupta and Kevin Zhai and Zhenyi Wang and Mubarak Shah},
url = {https://arxiv.org/abs/2603.25711
https://ai.ucf.edu/wp-content/uploads/2026/09/Seeing-to-Ground-ECCV.jpg
https://youtu.be/60Acip-ylyM?si=WtRA0tTr-quzOOCO},
year = {2026},
date = {2026-09-07},
urldate = {2026-09-07},
booktitle = {European Conference on Computer Vision (ECCV)},
publisher = {Springer},
abstract = {Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.},
keywords = {ECCV, Hallucination Mitigation, Multimodal LLMs},
pubstate = {published},
tppubtype = {conference}
}
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.