Designing for GPU Farm Reliability and Rolling Updates

Andy Pernsteiner is the Field CTO at VAST Data, working on large-scale AI infrastructure, serverless compute near data, and the rollout of VAST’s AI Operating System.

The GPU Uptime Battle // MLOps Podcast #346 with Andy Pernsteiner, Field CTO of VAST Data.Huge thanks to VAST Data for supporting this episode!

Join the Community:

https://go.mlops.community/YTJoinIn

Get the newsletter:

https://go.mlops.community/YTNewsletter

// Abstract

Most AI projects don’t fail because of bad models; they fail because of bad data plumbing. Andy Pernsteiner joins the podcast to talk about what it actually takes to build production-grade AI systems that aren’t held together by brittle ETL scripts and data copies. He unpacks why unifying data - rather than moving it - is key to real-time, secure inference, and how event-driven, Kubernetes-native pipelines are reshaping the way developers build AI applications. It’s a conversation about cutting out the complexity, keeping data live, and building systems smart enough to keep up with your models.

// Bio

Andy is the Field Chief Technology Officer at VAST, helping customers build, deploy, and scale some of the world’s largest and most demanding computing environments.

Andy has spent the past 15 years focused on supporting and building large-scale, high-performance data platform solutions. From humble beginnings as an escalations engineer at pre-IPO Isilon, to leading a team of technical Ninjas at MapR, he’s consistently been in the frontlines solving some of the toughest challenges that customers face when implementing Big Data Analytics and next-generation AI solutions.

// Related Links

Website: www.vastdata.com

https://www.youtube.com/watch?v=HYIEgFyHaxk

https://www.youtube.com/watch?v=RyDHIMniLro

The Mom Test by Rob Fitzpatrick: https://www.momtestbook.com/

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community

[https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Andy on LinkedIn: /andypernsteiner

Timestamps:

[00:00] Prototype to production gap

[00:21] AI expectations vs reality

[03:00] Prototype vs production costs

[07:47] Technical debt awareness

[10:13] The Mom Test

[15:40] Chaos engineering

[22:25] Data messiness reflection

[26:50] Small data value

[30:53] Platform engineer mindset shift

[34:26] Gradient description comparison

[38:12] Empathy in MLOps

[45:48] Empathy in Engineering

[51:04] GPU clusters rolling updates

[1:03:14] Checkpointing strategy comparison

[1:09:44] Predictive vs Generative AI

[1:17:51] On Growth, Community, and New Directions

[1:24:21] UX of agents

[1:32:05] Wrap up