Arxiv paper - EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

In this episode, we discuss EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents by Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang. The paper presents **EMBODIEDBENCH**, a comprehensive benchmark with 1,128 tasks across four environments to evaluate vision-driven embodied agents based on Multi-modal Large Language Models (MLLMs). It assesses key capabilities such as commonsense reasoning, spatial awareness, and long-term planning through six specialized subsets. Evaluations of 13 MLLMs revealed that while these models perform well on high-level tasks, they struggle with low-level manipulations, highlighting significant challenges and guiding future advancements in embodied agent development.