Latency vs Capacity Trade-offs for Serving Models

They debate whether lowering inference latency or increasing capacity matters more for user experience and model adoption.

Transcript

chevron_right

Play full episode

chevron_right

Transcript

Episode notes

My guest today is Dylan Patel. Dylan is the founder and CEO of SemiAnalysis. At SemiAnalysis Dylan tracks the semiconductor supply chain and AI infrastructure buildout with unmatched granularity—literally watching data centers get built through satellite imagery and mapping hundreds of billions in capital flows. Our conversation explores the massive industrial buildout powering AI, from the strategic chess game between OpenAI, Nvidia, and Oracle to why we're still in the first innings of post-training and reinforcement learning. Dylan explains infrastructure realities like electrician wages doubling and companies using diesel truck engines for emergency power, while making a sobering case about US-China competition and why America needs AI to succeed. We discuss his framework for where value will accrue in the stack, why traditional SaaS economics are breaking down under AI's high cost of goods sold, and which hardware bottlenecks matter most. This is one of the most comprehensive views of the physical reality underlying the AI revolution you'll hear anywhere. Please enjoy my conversation with Dylan Patel.

For the full show notes, transcript, and links to mentioned content, check out the episode page ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠here⁠⁠⁠⁠⁠⁠.⁠⁠⁠⁠⁠⁠⁠⁠

-----

This episode is brought to you by⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Ramp⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠. Ramp’s mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Ramp.com/invest⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ to sign up for free and get a $250 welcome bonus.

–

This episode is brought to you by⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Ridgeline⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠. Ridgeline has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Head to⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ridgelineapps.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ to learn more about the platform.

–

This episode is brought to you by⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ AlphaSense⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠. AlphaSense has completely transformed the research process with cutting-edge AI technology and a vast collection of top-tier, reliable business content. Invest Like the Best listeners can get a free trial now at⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ Alpha-Sense.com/Invest⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ and experience firsthand how AlphaSense and Tegus help you make smarter decisions faster.

-----

Editing and post-production work for this episode was provided by The Podcast Consultant (⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://thepodcastconsultant.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠).

Show Notes:

(00:00:00) Welcome to Invest Like the Best

(00:05:12) The AI Infrastructure Buildout

(00:08:25) Scaling AI Models and Compute Needs

(00:11:44) Reinforcement Learning and AI Training

(00:14:07) The Future of AI and Compute

(00:17:47) AI in Practical Applications

(00:22:29) The Importance of Data and Environments in AI Training

(00:29:45) Human Analogies in AI Development

(00:40:34) The Challenge of Infinite Context in AI Models