2026
Mollah, Sabbir; Gupta, Rohit; Sirnam, Swetha; Liu, Qingyang; Munir, Ahnaf; Shah, Mubarak
The Telephone Game: Evaluating Semantic Drift in Unified Models Conference
European Conference on Computer Vision (ECCV), Springer, 2026.
Abstract | Tags: ECCV, Model Evaluation, Vision-Language Models | Links:
@conference{mollah2026telephonegameb,
title = {The Telephone Game: Evaluating Semantic Drift in Unified Models},
author = {Sabbir Mollah and Rohit Gupta and Swetha Sirnam and Qingyang Liu and Ahnaf Munir and Mubarak Shah},
url = {https://arxiv.org/abs/2509.04438
https://ai.ucf.edu/wp-content/uploads/2026/09/Poster__Telephone_Game.pdf
https://youtu.be/1y_Pn6xD0fw?si=Oab_zk-25L591fLO},
year = {2026},
date = {2026-09-07},
urldate = {2026-09-07},
booktitle = {European Conference on Computer Vision (ECCV)},
publisher = {Springer},
abstract = {Employing a single, unified model (UM) for both visual understanding (image-to-text: I2T) and visual generation (text-to-image: T2I) has opened a new direction in Visual Language Model (VLM) research. While UMs can also support broader unimodal tasks (e.g., text-to-text, image-to-image), we focus on the core cross-modal pair T2I and I2T. Existing evaluation benchmarks consider these capabilities in isolation: FID and GenEval for T2I, and benchmarks such as MME, MMBench for I2T. These isolated single-pass metrics do not reveal cross-consistency: whether a model that "understands" a concept can also "render" it, nor whether semantic meaning is preserved when cycling between image and text modalities. To address this, we introduce the Semantic Drift Protocol (SDP) for Unified Models, a cyclic evaluation protocol that alternates I2T and T2I over multiple generations to quantify semantic drift. We propose two metrics: (i) Mean Cumulative Drift (MCD), an embedding-based measure of overall semantic drift; and (ii) Multi-Generation GenEval (MGG), an object-level compliance score extending GenEval. To assess generalization beyond COCO dataset, which is widely used in training; we create a new benchmark Nocaps+Docci400, sampled from NoCaps and DOCCI and evaluated on seven recent models. SDP reveals substantial variation in cross-modal stability: some models like BAGEL maintain semantic meaning over many alternations, whereas others like VILA-U drift quickly despite strong single-pass scores. Our results highlight SDP as a necessary complement to standard I2T and T2I evaluations. Code is available at this https URL},
keywords = {ECCV, Model Evaluation, Vision-Language Models},
pubstate = {published},
tppubtype = {conference}
}
Employing a single, unified model (UM) for both visual understanding (image-to-text: I2T) and visual generation (text-to-image: T2I) has opened a new direction in Visual Language Model (VLM) research. While UMs can also support broader unimodal tasks (e.g., text-to-text, image-to-image), we focus on the core cross-modal pair T2I and I2T. Existing evaluation benchmarks consider these capabilities in isolation: FID and GenEval for T2I, and benchmarks such as MME, MMBench for I2T. These isolated single-pass metrics do not reveal cross-consistency: whether a model that "understands" a concept can also "render" it, nor whether semantic meaning is preserved when cycling between image and text modalities. To address this, we introduce the Semantic Drift Protocol (SDP) for Unified Models, a cyclic evaluation protocol that alternates I2T and T2I over multiple generations to quantify semantic drift. We propose two metrics: (i) Mean Cumulative Drift (MCD), an embedding-based measure of overall semantic drift; and (ii) Multi-Generation GenEval (MGG), an object-level compliance score extending GenEval. To assess generalization beyond COCO dataset, which is widely used in training; we create a new benchmark Nocaps+Docci400, sampled from NoCaps and DOCCI and evaluated on seven recent models. SDP reveals substantial variation in cross-modal stability: some models like BAGEL maintain semantic meaning over many alternations, whereas others like VILA-U drift quickly despite strong single-pass scores. Our results highlight SDP as a necessary complement to standard I2T and T2I evaluations. Code is available at this https URL