The Information Bottleneck

Jonas Geiping, a machine learning researcher at the ELLIS Institute and Max Planck Institute, explores the fascinating world of recurrent-depth models and latent reasoning. He discusses how these models can enhance AI's reasoning capabilities, especially in complex tasks like math and coding. The conversation also delves into challenges in model development, the importance of interpretability and safety in AI, and the future of scalable algorithms. With practical advice for budding researchers, Jonas sheds light on Tübingen as an emerging hub for machine learning innovation.

Join Jack Morris, a PhD student at Cornell and creator of the TextAttack library, as he dives into the intriguing world of adversarial examples in language models. Jack discusses the evolution of TextAttack, the complexities of open-source AI, and the security implications of inversion attacks. He highlights the Platonic representation hypothesis and its impact on understanding model embeddings. Plus, he connects the dots between compression in language models and their efficiency. Get ready for a fascinating exploration of the future of AI!

Randall Balestriero, an assistant professor at Brown University specializing in representation learning, dives deep into Joint Embedding Predictive Architectures (JEPA). He explains how JEPA learns data representations without reconstruction, focusing on meaningful features while compressing irrelevant details. The discussion covers the challenges of model collapse, prediction tasks shaping feature learning, and the implications for AGI benchmarks. Balestriero also shares insights on evaluating JEPA models, the role of latent variables, and the growing opportunity in JEPA research.

Michael Bronstein, a Professor of AI at Oxford and scientific director at AITHYRA, dives deep into the realm of geometric deep learning. He discusses the debate between small neural networks and scaling, emphasizing the significance of geometric structures in enhancing AI efficiency. The conversation spans challenges in building effective tabular models and the role of instruction tuning. Michael also explores exciting applications in drug design, detailing how AI can revolutionize protein modeling and therapeutic strategies. His insights bridge the gap between theory and practical innovations in science.

Guest Tal Kachman is an Assistant Professor at Radboud University, specializing in the intersection of AI and natural sciences. He discusses the innovative role of self-driving labs in materials discovery and the impact of automation on high-throughput experiments. Tal elaborates on using neural ODEs to analyze chemical processes and recover unknown reaction rates. He also addresses the challenges of integrating physics with AI, emphasizing the significance of data quality and standardization in scientific research.

Ahmad Beirami, a former Google researcher, dives into the intricacies of reinforcement learning and its relevance to AI models. He highlights the evaluation challenges in AI research and argues for a shift towards deeper analysis rather than chasing small gains. Ahmad also critiques the current conference review system, revealing its strain and the issues it creates. Discussions include agent workflows, the implications of quantization, and the need for better methods in RL evaluation, all emphasizing the importance of integrating theoretical insights with empirical work.

In this discussion, Aran Nayebi, an Assistant Professor at Carnegie Mellon University specializing in computational neuroscience and AI, shares his insights on blending machine learning with brain science. He explores the evolution of neural networks and intrinsic motivation's role in AI development. The conversation dives into the nuances of cross-species modeling, the importance of lifelong learning, and how better brain-machine interfaces can be achieved through individualized data. Nayebi's fascinating zebrafish experiments reveal connections between model objectives and neural circuits.

We talked with Ariel Noyman, an urban scientist, working in the intersection of cities and technology. Ariel is a research scientist at the MIT Media Lab, exploring novel methods of urban modeling and simulation using AI. We discussed the potential of virtual environments to enhance urban design processes, the challenges associated with them, and the future of utilizing AI. Links:TravelAgent: Generative agents in the built environment - https://journals.sagepub.com/doi/10.1177/23998083251360458Ariel Neumann's websites -https://www.arielnoyman.com/https://www.media.mit.edu/people/noyman/overview/

We discussed the inference optimization technique known as Speculative Decoding with a world class researcher, expert, and ex-coworker of the podcast hosts: Nadav Timor.Papers and links:Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies, Timor et al, ICML 2025, https://arxiv.org/abs/2502.05202Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference, Timor et al, ICLR, 2025, https://arxiv.org/abs/2405.14105Fast Inference from Transformers via Speculative Decoding, Leviathan et al, 2022, https://arxiv.org/abs/2502.05202FindPDFs - https://huggingface.co/datasets/HuggingFaceFW/finepdfs

In this episode, Ravid and Allen discuss the evolving landscape of AI coding. They explore the rise of AI-assisted development tools, the challenges faced in software engineering, and the potential future of AI in creative fields. The conversation highlights both the benefits and limitations of AI in coding, emphasizing the need for careful consideration of its impact on the industry and society.Chapters00:00Introduction to AI Coding and Recent Developments03:10OpenAI's Paper on Hallucinations in LLMs06:03Critique of OpenAI's Research Approach08:50Copyright Issues in AI Training Data12:00The Value of Data in AI Training14:50Watermarking AI Generated Content17:54The Future of AI Investment and Market Dynamics20:49AI Coding and Its Impact on Software Development31:36The Evolution of AI in Software Development33:54Vibe Coding: The Future or a Fad?38:24Navigating AI Tools: Personal Experiences and Challenges41:53The Limitations of AI in Complex Coding Tasks46:52Security Vulnerabilities in AI-Generated Code50:28The Role of Human Intuition in AI-Assisted Coding53:28The Impact of AI on Developer Productivity56:53The Future of AI in Creative Fields