SF Compute: Commoditizing Compute
Latent Space: The AI Engineer Podcast
Navigating GPU Market Challenges
This chapter explores the financial hurdles companies face when integrating hardware and software, particularly in the GPU cloud sector. It highlights the emergence of a platform called SF Compute, designed for flexible GPU resource management, and discusses the fluctuations in GPU supply and demand. Additionally, the chapter addresses the dynamics of funding in AI startups and the impact of traditional pricing models on new innovators.
Evan Conrad, co-founder of SF Compute, joined us to talk about how they started as an AI lab that avoided bankruptcy by selling GPU clusters, why CoreWeave financials look like a real estate business, and how GPUs are turning into a commodities market.
Chapters:
00:00:05 - Introductions
00:00:12 - Introduction of guest Evan Conrad from SF Compute
00:00:12 - CoreWeave Business Model Discussion
00:05:37 - CoreWeave as a Real Estate Business
00:08:59 - Interest Rate Risk and GPU Market Strategy Framework
00:16:33 - Why Together and DigitalOcean will lose money on their clusters
00:20:37 - SF Compute's AI Lab Origins
00:25:49 - Utilization Rates and Benefits of SF Compute Market Model
00:30:00 - H100 GPU Glut, Supply Chain Issues, and Future Demand Forecast
00:34:00 - P2P GPU networks
00:36:50 - Customer stories
00:38:23 - VC-Provided GPU Clusters and Credit Risk Arbitrage
00:41:58 - Market Pricing Dynamics and Preemptible GPU Pricing Model
00:48:00 - Future Plans for Financialization?
00:52:59 - Cluster auditing and quality control
00:58:00 - Futures Contracts for GPUs
01:01:20 - Branding and Aesthetic Choices Behind SF Compute
01:06:30 - Lessons from Previous Startups
01:09:07 - Hiring at SF Compute
Chapters
- 00:00:00 Introduction and Background
- 00:00:58 Analysis of GPU Business Models
- 00:01:53 Challenges with GPU Pricing
- 00:02:48 Revenue and Scaling with GPUs
- 00:03:46 Customer Sensitivity to GPU Pricing
- 00:04:44 Core Weave's Business Strategy
- 00:05:41 Core Weave's Market Perception
- 00:06:40 Hyperscalers and GPU Market Dynamics
- 00:07:37 Financial Strategies for GPU Sales
- 00:08:35 Interest Rates and GPU Market Risks
- 00:09:30 Optimal GPU Contract Strategies
- 00:10:27 Risks in GPU Market Contracts
- 00:11:25 Price Sensitivity and Market Competition
- 00:12:21 Market Dynamics and GPU Contracts
- 00:13:18 Hyperscalers and GPU Market Strategies
- 00:14:15 Nvidia and Market Competition
- 00:15:12 Microsoft's Role in GPU Market
- 00:16:10 Challenges in GPU Market Dynamics
- 00:17:07 Economic Realities of the GPU Market
- 00:18:03 Real Estate Model for GPU Clouds
- 00:18:59 Price Sensitivity and Chip Design
- 00:19:55 SF Compute's Beginnings and Challenges
- 00:20:54 Navigating the GPU Market
- 00:21:54 Pivoting to a GPU Cloud Provider
- 00:22:53 Building a GPU Market
- 00:23:52 SF Compute as a GPU Marketplace
- 00:24:49 Market Liquidity and GPU Pricing
- 00:25:47 Utilization Rates in GPU Markets
- 00:26:44 Brokerage and Market Flexibility
- 00:27:42 H100 Glut and Market Cycles
- 00:28:40 Supply Chain Challenges and GPU Glut
- 00:29:35 Future Predictions for the GPU Market
- 00:30:33 Speculations on Test Time Inference
- 00:31:29 Market Demand and Test Time Inference
- 00:32:26 Open Source vs. Closed AI Demand
- 00:33:24 Future of Inference Demand
- 00:34:24 Peer-to-Peer GPU Markets
- 00:35:17 Decentralized GPU Market Skepticism
- 00:36:15 Redesigning Architectures for New Markets
- 00:37:14 Supporting Grad Students and Startups
- 00:38:11 Successful Startups Using SF Compute
- 00:39:11 VCs and GPU Infrastructure
- 00:40:09 VCs as GPU Credit Transformators
- 00:41:06 Market Timing and GPU Infrastructure
- 00:42:02 Understanding GPU Pricing Dynamics
- 00:43:01 Market Pricing and Preemptible Compute
- 00:43:55 Price Volatility and Market Optimization
- 00:44:52 Customizing Compute Contracts
- 00:45:50 Creating Flexible Compute Guarantees
- 00:46:45 Financialization of GPU Markets
- 00:47:44 Building a Spot Market for GPUs
- 00:48:40 Auditing and Standardizing Clusters
- 00:49:40 Ensuring Cluster Reliability
- 00:50:36 Active Monitoring and Refunds
- 00:51:33 Automating Customer Refunds
- 00:52:33 Challenges in Cluster Maintenance
- 00:53:29 Remote Cluster Management
- 00:54:29 Standardizing Compute Contracts
- 00:55:28 Unified Infrastructure for Clusters
- 00:56:24 Creating a Commodity Market for GPUs
- 00:57:22 Futures Market and Risk Management
- 00:58:18 Reducing Risk with GPU Futures
- 00:59:14 Stabilizing the GPU Market
- 01:00:10 SF Compute's Anti-Hype Approach
- 01:01:07 Calm Branding and Expectations
- 01:02:07 Promoting San Francisco's Beauty
- 01:03:03 Design Philosophy at SF Compute
- 01:04:02 Artistic Influence on Branding
- 01:05:00 Past Projects and Burnout
- 01:05:59 Challenges in Building an Email Client
- 01:06:57 Persistence and Iteration in Startups
- 01:07:57 Email Market Challenges
- 01:08:53 SF Compute Job Opportunities
- 01:09:53 Hiring for Systems Engineering
- 01:10:50 Financial Systems Engineering Role
- 01:11:50 Conclusion and Farewell
The AI-powered Podcast Player
