Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding
A training-free decoding method called MGAP aims to reduce MLLM hallucinations without harming coherence. It selectively dampens harmful language priors while preserving useful semantic structure.
Intelligence analysis by GPT-5.4 Mini
The paper argues that MLLMs do not just suffer from too much language bias; some of that bias is useful. MGAP tries to separate the helpful part from the harmful part and suppress only the latter during decoding.
A smart picture-answering robot sometimes guesses from habit instead of looking carefully, like blurting out “dog” just because dogs are common. This paper tries to turn down only the bad guessing part while leaving the helpful thinking part alone.
Analysis
Problem
Multimodal large language models often describe objects that are not actually present in the image. The paper says a common cause is over-reliance on language priors, where the model leans on likely words instead of visual evidence.
What prior fixes miss
Recent training-free decoding methods try to curb hallucinations by penalizing language priors. The authors argue that this is too blunt: language priors are not always bad, because they can also help the model stay coherent and semantically grounded. If the model suppresses them indiscriminately, performance can drop. The paper calls this failure mode Manifold Departure.
MGAP
To address that, the paper proposes Manifold-Guided Adaptive Projection (MGAP). It first builds a language-prior subspace from blind hidden states using SVD. During decoding, each multimodal hidden state is projected into that subspace, then a consistency-aware gate decides how much of the projected prior component should be attenuated.
The key idea is selective correction: MGAP reduces the part of the representation tied to harmful prior bias while largely preserving the orthogonal semantic components. In the paper’s framing, this keeps the model closer to its semantic manifold instead of pushing it off course.
Results
The authors report experiments on POPE and CHAIR. According to the paper, MGAP beats prior decoding baselines and suppresses hallucinations more strongly without sacrificing coherence. That makes it a training-free option for improving trustworthiness in MLLM decoding.
Key points
- The paper argues that language priors in MLLMs can be both helpful and harmful, depending on visual alignment.
- It introduces the term Manifold Departure for cases where blunt prior suppression damages semantic structure.
- MGAP builds a language-prior subspace with SVD and adaptively attenuates only the projected prior component.
- The authors report stronger hallucination suppression on POPE and CHAIR without losing coherence.
If MGAP generalizes, it could make multimodal models less likely to invent objects while keeping answers readable and consistent. Because it is training-free, it may be easier to apply than methods that require retraining a whole model.
The method depends on how well its subspace and gate separate helpful priors from harmful ones, which may not transfer equally across models or tasks. If that separation is imperfect, the same kind of performance loss the paper warns about could still show up in other settings.
