Preference alignment has become crucial for enhancing Large Language Models (LLMs), but its impact on Multimodal Large Language Models (MLLMs) remains underexplored. MLLMs face hallucination challenges not only through incorrect facts but also through responses inconsistent with image content. Alignment aims to make MLLMs rely more on visual information. Many recent works introduce preference datasets and apply methods like Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO), yet variations in datasets, base models, and algorithms make it unclear which components drive improvements.

The authors independently analyze each aspect of multimodal preference alignment. They categorize algorithms into offline (e.g., DPO) and online (e.g., online-DPO) groups and demonstrate that combining offline and online methods can improve model performance in certain scenarios. They review several published multimodal preference datasets and discuss how dataset construction details affect downstream performance.

Based on these insights, the authors introduce Bias-Driven Hallucination Sampling (BDHS), a novel method for creating multimodal preference data that requires neither additional annotation nor external models. BDHS achieves competitive performance compared to previously published alignment work across a range of benchmarks for multimodal models.