AI + a16z Inside vLLM: The Engine Powering Open-Source AI
55 snips
Aug 6, 2026 
Simon Mo, co-founder and CEO of Inferact and lead maintainer of vLLM, builds open-source inference for running large language models at scale. He discusses how vLLM bridges models and GPUs, the shift from research project to production infrastructure, model release co-design, open-weight economics like Kimi K3, performance tuning and fast mode, and why open inference is now essential.
AI Snips
Chapters
Transcript
Episode notes
Inference Software And Hardware Are Co-Designed
- vLLM works closely with hardware vendors and is often used as a benchmark for new chips.
- Simon Mo lists NVIDIA, AMD, Google, Amazon and Intel as partners ensuring new chips run vLLM well.
How Mistral's Weekend Scramble Became Day Zero Support
- Early open-model releases were chaotic; Mistral initially dropped a torrent and users scrambled to run it.
- Simon Mo recounts a weekend working with Mistral's team to get inference support in vLLM and then announcing a working runtime on Monday.
Run Models Yourself To Guarantee SLAs
- Choose open-weight models when you need control over performance, SLAs, and data retention.
- Simon Mo explains voice-agent companies run models themselves to guarantee response times and monitor the whole hardware/software stack.