Invest Like the Best with Patrick O'Shaughnessy

Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]

1486 snips
Aug 25, 2026
Neil Movva, founder of Sail, builds a 'token factory' to drive inference cost way down. He walks through optimizing software, chips, and power for long-running AI agents. Short talks cover the throughput vs latency tradeoff, scavenging underused chips and intermittent power, and why ultra-cheap tokens could reshape always-on agents.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
ANECDOTE

NVIDIA's Speed Of Light Ethos

  • Neil worked on GPU kernels at NVIDIA and learned their culture of chasing
INSIGHT

Throughput Versus Latency Is A Fundamental Tradeoff

  • GPUs trade off throughput for latency: batching many users raises utilization but increases per-request time.
  • Neil compares it to a bus versus private car: batching is efficient but slower for individual requests.
INSIGHT

NVLink Optimizes Latency Not Cheap Throughput

  • NVLink and multi-GPU sharding optimize for minimum latency by slicing large matrix multiplies across GPUs.
  • That lowers single-request latency but uses more hardware sublinearly; it's essential for low-latency services but costly for throughput-first stacks.
Get the Snipd Podcast app to discover more snips from this episode
Get the app