AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
4min snip
Episode 29: Jim Fan, NVIDIA, on foundation models for embodied agents, scaling data, and why prompt engineering will become irrelevant
Generally Intelligent
The VPT Approach: Training AI Agents in Minecraft
OpenAI has a different approach to training agents using human data from Minecraft/nThe first step is to hire humans to play Minecraft and record their actions/nThe recorded human data is used to train an inverse dynamics model/nThe inverse dynamics model can label actions in YouTube videos with no explicit action data/nKeyboard and mouse actions are important for aligning YouTube video actions with game actions/nUsing the inverse dynamics model, more videos can be labeled than with human data alone/nA behavior cloning agent trained on the labeled videos can perform well on long horizon tasks/nVPT approach allows for direct imitation from tutorial videos without relying on rewards/nCollecting human data can be costly and may limit applicability to other domains/nThe VPT approach does not utilize language transcripts, but language could play a significant role/nLanguage can be used for prompting and grounding in embodied agent setups/nBoth VPT and MindOjo/MindClip approaches are complementary and important for success in Minecraft
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode