arxiv preprint - LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

In this episode, we discuss LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning by Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, Roei Herzig. The paper introduces LLARVA, a model improved with a novel instruction-tuning method to unify various robotic tasks using structured prompts. The model utilizes 2-D visual traces to better align vision and action spaces, pre-trained on 8.5M image-visual trace pairs from the Open X-Embodiment dataset. Experiments on the RLBench simulator and a physical robot demonstrate that LLARVA outperforms several baselines and generalizes well across different robotic environments.