Building Scalable ML Systems on Kubernetes

5 snips

Aug 15, 2024

Tammer Saleh, founder of SuperOrbital and an expert in scalable machine learning systems, discusses the advantages and challenges of using Kubernetes for ML workloads. He highlights the importance of model tracking and versioning within containerized environments. The conversation touches on the necessity of a unified API for collaboration across teams and the evolving imperfections of Kubernetes in stateful ML contexts. Tammer also shares insights on future innovations and best practices for teams navigating the complexities of machine learning on Kubernetes.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

From Server Rooms to Heroku

Tammer Saleh's cloud journey began with physical servers and evolved through Engine Yard and Heroku.
He recounts the initial skepticism towards Heroku's cloud IDE, which later influenced his perspective on cloud platforms.

INSIGHT

The 12-Factor Model's Impact

Heroku's 12-factor model, while initially mocked, was a pivotal moment in cloud platform thinking.
It emphasized treating applications as good Unix processes, a principle that influenced Saleh's approach to Kubernetes.

INSIGHT

Kubernetes: More Than Just Containers

Kubernetes runs containerized workloads at scale in production, offering more than just container orchestration.
Its key strength lies in embodying Google's SRE best practices and providing a powerful, unified API for automation.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the AI Engineering podcast, host Tobias Macy interviews Tammer Saleh, founder of SuperOrbital, about the potentials and pitfalls of using Kubernetes for machine learning workloads. The conversation delves into the specific needs of machine learning workflows, such as model tracking, versioning, and the use of Jupyter Notebooks, and how Kubernetes can support these tasks. Tammer emphasizes the importance of a unified API for different teams and the flexibility Kubernetes provides in handling various workloads. Finally, Tammer offers advice for teams considering Kubernetes for their machine learning workloads and discusses the future of Kubernetes in the ML ecosystem, including areas for improvement and innovation.
Announcements

Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems
Your host is Tobias Macey and today I'm interviewing Tammer Saleh about the potentials and pitfalls of using Kubernetes for your ML workloads.

Interview

Introduction
How did you get involved in Kubernetes?
For someone who is unfamiliar with Kubernetes, how would you summarize it?
For the context of this conversation, can you describe the different phases of ML that we're talking about?
Kubernetes was originally designed to handle scaling and distribution of stateless processes. ML is an inherently stateful problem domain. What challenges does that add for K8s environments?
What are the elements of an ML workflow that lend themselves well to a Kubernetes environment?
How much Kubernetes knowledge does an ML/data engineer need to know to get their work done?
What are the sharp edges of Kubernetes in the context of ML projects?
What are the most interesting, unexpected, or challenging lessons that you have learned while working with Kubernetes?
When is Kubernetes the wrong choice for ML?
What are the aspects of Kubernetes (core or the ecosystem) that you are keeping an eye on which will help improve its utility for ML workloads?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for ML workloads today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA