PhD Value and Learning by Doing

In this episode, we host Jonas Geiping from ELLIS Institute & Max-Planck Institute for Intelligent Systems, Tübingen AI Center, Germany. We talked about his broad research on Recurrent-Depth Models and latent reasoning in large language models (LLMs). We talked about what these models can and can't do, what are the challenges and next breakthroughs in the field, world models, and the future of developing better models. We also talked about safety and interpretability, and the role of scaling laws in AI development.

Chapters

00:00 Introduction and Guest Introduction

01:03 Peer Review in Preprint Servers

06:57 New Developments in Coding Models

09:34 Open Source Models in Europe

11:00 Dynamic Layers in LLMs

26:05 Training Playbook Insights

30:05 Recurrent Depth Models and Reasoning Tasks

43:59 Exploring Recursive Reasoning Models

46:46 The Role of World Models in AI

48:41 Innovations in AI Training and Simulation

50:39 The Promise of Recurrent Depth Models

52:34 Navigating the Future of AI Algorithms

54:44 The Bitter Lesson of AI Development

59:11 Advising the Next Generation of Researchers

01:06:42 Safety and Interpretability in AI Models

01:10:46 Scaling Laws and Their Implications

01:16:19 The Role of PhDs in AI Research

Links and paper:

• Jonas' website - https://jonasgeiping.github.io/
• Scaling up test-time compute with latent reasoning: A recurrent depth approach - https://arxiv.org/abs/2502.05171
• The Smol Training Playbook: The Secrets to Building World-Class LLMs - https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook
• VaultGemma: A Differentially Private Gemma Model - https://arxiv.org/abs/2510.15001

Music:

“Kid Kodi” — Blue Dot Sessions — via Free Music Archive — CC BY-NC 4.0.

“Palms Down” — Blue Dot Sessions — via Free Music Archive — CC BY-NC 4.0.

Changes: trimmed