Latent Space: The AI Engineer Podcast

Lubna Ben-Alau, an AI researcher at Hugging Face, dives into the dynamic world of synthetic data and small language models. She discusses how 2024 saw a remarkable surge in synthetic data applications, with notable contributions like Apple's Rephrasing the Web and Hugging Face's Cosmopedia. Lubna emphasizes the transformative impact of synthetic data on model performance and diversity. The conversation also touches on the evolution of small models, highlighting their efficiency, improved privacy, and specialized applications for on-device use.

Dan Fu, an AI researcher soon to join UCSD, and Eugene Cheah, CEO of Featherless AI, delve into the future of post-transformer architectures. They discuss innovations like RWKV and state-space models, highlighting their collaborative and open-source nature. The duo examines the challenges of multilingual training and computational efficiency, while also exploring advancements in non-transformer models like Mamba and Jamba. Tune in for insights on scaling models, 'infinite context,' and how new architectures are reshaping the AI landscape!

Luca Soldaini, a research scientist at the Allen Institute for AI, and Sophia Yang, head of Developer Relations at Mistral AI, dive into the explosive rise of open models in 2024. They discuss breakthrough models like Llama 3 and the MOE model, highlighting the competitive dynamics in AI. Key challenges such as regulatory hurdles and limited training data access are explored. The conversation also emphasizes the need for collaboration and open-source methodologies to foster innovation in a rapidly evolving landscape.

In this engaging discussion, Isaac Robinson and Peter Robicheaux from Roboflow share insights on the latest trends and groundbreaking papers in computer vision for 2024. They highlight the shift towards video-based models like 'Sora' and advancements in real-time object detection. Vik Korrapati, founder of Moondream, presents challenges in developing vision language models and introduces a compact, pruned model. Together, they explore how these innovations can reshape the landscape of computer vision and enhance pre-trained model efficiencies.

The opening keynote at NeurIPS 2024 dives into the evolving landscape of AI startups and investment trends. Experts discuss transformative developments and the shift towards open-source models. They analyze advancements in video modalities and challenges in scaling businesses. The conversation touches on integrating multimodal data in enterprises, the impact of declining AI costs, and the growing interest in consumer-focused AI startups. Exciting insights highlight the future opportunities for innovation and competition in the AI industry.

Varun Mohan, CEO of Codeium AI, leads the charge in revolutionizing developer tools with his innovative IDE, Windsurf. He discusses the challenges faced by traditional platforms like VS Code and introduces the advanced features of Windsurf, such as its intuitive interface and enhanced user experience. Varun reveals insights from user feedback post-launch and emphasizes the importance of balancing automation with human input. He also shares Codeium's evolution and strategies for catering to both individual developers and enterprises while maintaining their commitment to free access.

Regular tickets are now sold out for Latent Space LIVE! at NeurIPS! We have just announced our last speaker and newest track, friend of the pod Nathan Lambert who will be recapping 2024 in Reasoning Models like o1! We opened up a handful of late bird tickets for those who are deciding now — use code DISCORDGANG if you need it. See you in Vancouver!We’ve been sitting on our ICML recordings for a while (from today’s first-ever SOLO guest cohost, Brittany Walker), and in light of Sora Turbo’s launch (blogpost, tutorials) today, we figured it would be a good time to drop part one which had been gearing up to be a deep dive into the state of generative video worldsim, with a seamless transition to vision (the opposite modality), and finally robots (their ultimate application).Sora, Genie, and the field of Generative Video World SimulatorsBill Peebles, author of Diffusion Transformers, gave his most recent Sora talk at ICML, which begins our episode:* William (Bill) Peebles - SORA (slides)Something that is often asked about Sora is how much inductive biases were introduced to achieve these results. Bill references the same principles brought by Hyung Won Chung from the o1 team - “sooner or later those biases come back to bite you”.We also recommend these reads from throughout 2024 on Sora.* Lilian Weng’s literature review of Video Diffusion Models* Sora API leak* Estimates of 100k-700k H100s needed to serve Sora (not Turbo)* Artist guides on using Sora for professional storytellingGoogle DeepMind had a remarkably strong presence at ICML on Video Generation Models, winning TWO Best Paper awards for:* Genie: Generative Interactive Environments (covered in oral, poster, and workshop)* VideoPoet: A Large Language Model for Zero-Shot Video Generation (see website)We end this part by taking in Tali Dekel’s talk on The Future of Video Generation: Beyond Data and Scale.Part 2: Generative Modeling and DiffusionSince 2023, Sander Dieleman’s perspectives (blogpost, tweet) on diffusion as “spectral autoregression in the frequency domain” while working on Imagen and Veo have caught the public imagination, so we highlight his talk:* Wading through the noise: an intuitive look at diffusion modelsThen we go to Ben Poole for his talk on Inferring 3D Structure with 2D Priors, including his work on NeRFs and DreamFusion:Then we investigate two flow matching papers - one from the Flow Matching co-authors - Ricky T. Q. Chen (FAIR, Meta)And how it is implemented in Stable Diffusion 3 with Scaling Rectified Flow Transformers for High-Resolution Image Synthesis Our last hit on Diffusion is a couple of oral presentations on speech, which we leave you to explore via our audio podcast* NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models* Speech Self-Supervised Learning Using Diffusion Model Synthetic DataPart 3: VisionThe ICML Test of Time winner was DeCAF, which Trevor Darrell notably called “the OG vision foundation model”.Lucas Beyer’s talk on “Vision in the age of LLMs — a data-centric perspective” was also well received online, and he talked about his journey from Vision Transformers to PaliGemma.We give special honorable mention to MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark.Part 4: Reinforcement Learning and RoboticsWe segue vision into robotics with the help of Ashley Edwards, whose work on both the Gato and the Genie teams at Deepmind is summarized in Learning actions, policies, rewards, and environments from videos alone.Brittany highlighted two poster session papers:* Behavior Generation with Latent Actions* We also recommend Lerrel Pinto’s On Building General-Purpose Robots* PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMsHowever we must give the lion’s share of space to Chelsea Finn, now founder of Physical Intelligence, who gave FOUR talks on* "What robots have taught me about machine learning"* developing robot generalists* robots that adapt autonomously* how to give feedback to your language model* special mention to PI colleague Sergey Levine on Robotic Foundation ModelsWe end the podcast with a position paper that links generative environments and RL/robotics: Automatic Environment Shaping is the Next Frontier in RL.Timestamps* [00:00:00] Intros* [00:02:43] Sora - Bill Peebles* [00:44:52] Genie: Generative Interactive Environments* [01:00:17] Genie interview* [01:12:33] VideoPoet: A Large Language Model for Zero-Shot Video Generation* [01:30:51] VideoPoet interview - Dan Kondratyuk* [01:42:00] Tali Dekel - The Future of Video Generation: Beyond Data and Scale.* [02:27:07] Sander Dieleman - Wading through the noise: an intuitive look at diffusion models* [03:06:20] Ben Poole - Inferring 3D Structure with 2D Priors* [03:30:30] Ricky Chen - Flow Matching* [04:00:03] Patrick Esser - Stable Diffusion 3* [04:14:30] NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models* [04:27:00] Speech Self-Supervised Learning Using Diffusion Model Synthetic Data* [04:39:00] ICML Test of Time winner: DeCAF* [05:03:40] Lucas Beyer: “Vision in the age of LLMs — a data-centric perspective”* [05:42:00] Ashley Edwards: Learning actions, policies, rewards, and environments from videos alone.* [06:03:30] Behavior Generation with Latent Actions interview* [06:09:52] Chelsea Finn: "What robots have taught me about machine learning"* [06:56:00] Position: Automatic Environment Shaping is the Next Frontier in RL Get full access to Latent.Space at www.latent.space/subscribe

In this discussion, Itamar Friedman, CEO of Codo, and Eric Simons, CEO of StackBlitz, dive into the rapidly evolving world of AI-powered software development. They explore the impressive rise of Bolt.new, achieving over $8 million ARR in just two months. The duo highlights the transformation of StackBlitz and the role of AI in streamlining code generation. They also discuss challenges in platform development, the significance of user experience, and the exciting tools being developed to empower non-engineers in web development.

Erik Schluntz, a member of technical staff at Anthropic and former CTO of Cobalt Robotics, dives deep into the advancements of Claude 3.5 Sonnet. He discusses how this model transformed coding agents and tackled challenging benchmarks like SWE-Bench, improving performance dramatically. Erik also explores the complexities of building multimodal tools and the importance of user-friendly design. He shares insights on enhancing AI memory, the significance of real-world applications, and the future of cloud technology in automation.

Lin Qiao, Co-founder and CEO of Fireworks AI, shares insights into the dynamic world of open-source AI. She discusses the challenges and growth of her startup, emphasizing adaptability in a volatile environment. Lin highlights the advantages of open-source over closed source, arguing for its superior scalability and innovation potential. The conversation also delves into the evolution of PyTorch and the concept of Compound AI, showcasing how diverse modalities can enhance user experiences. Ultimately, Lin stresses the importance of community feedback in driving AI advancements.