RoboPapers Ep#19 Learning to Drive from a World Model
Jul 13, 2025
In this engaging discussion, Harald Schäfer leads the autonomy team at Comma AI, sharing insights from his eight-year journey in robotics. He dives into groundbreaking advancements in self-driving technology, emphasizing data-driven learning and world models. The conversation covers the challenges of developing versatile systems for various car models and innovative simulation strategies. Harald also explores the trade-offs in world model training, the importance of harnessing human-driven data, and the commitment to open-source innovations in automotive AI that could revolutionize user experiences.
AI Snips
Chapters
Transcript
Episode notes
Long Quest to End-to-End Driving
- Harald Schäfer spent about eight years developing an on-policy training method with simulation for Comma AI's autonomous driving system.
- The method is deployed in their product OpenPilot, used by real people for partial autonomy on highways.
Limits of Reprojection Simulator
- The original reprojection simulator assumes a static scene and fails to model reactions of other agents to the ego car's actions.
- Artifacts in depth and occlusion cause the model to cheat by learning these instead of actual driving behaviors.
Future Anchoring Enables Recovery
- Anchoring simulation to both past context and future real data enables world models to predict reasonable recovery trajectories.
- This breakthrough eliminates relying on classical control and better handles deviations during simulation.