Arxiv paper - Calibrated Multi-Preference Optimization for Aligning Diffusion Models

In this episode, we discuss Calibrated Multi-Preference Optimization for Aligning Diffusion Models by Kyungmin Lee, Xiaohang Li, Qifei Wang, Junfeng He, Junjie Ke, Ming-Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, Yinxiao Li. The paper introduces Calibrated Preference Optimization (CaPO), a new method for aligning text-to-image diffusion models using multiple reward models without requiring expensive human-annotated data. CaPO calibrates general preferences by calculating expected win-rates against pretrained model samples and employs a frontier-based pair selection to handle multi-preference distributions effectively. Experimental evaluations on benchmarks like GenEval and T2I-Compbench show that CaPO consistently outperforms existing methods such as Direct Preference Optimization in both single and multi-reward scenarios.