Kubernetes Podcast from Google

Working Group Serving, with Yuan Tang and Eduardo Arango

Oct 31, 2024

Yuan Tang is a principal software engineer at Red Hat, focusing on OpenShift AI, and is a leader in Kubernetes WG Serving. Eduardo Arango, a software engineer at NVIDIA, specializes in making Kubernetes suitable for high-performance computing. They delve into the challenges of AI model serving, discussing startup times and Kubernetes API limitations. The conversation also covers orchestration complexities for large language models and highlights innovative solutions like Model Mesh to optimize multi-host environments. Engagement and collaboration in Kubernetes working groups are urged for community-driven advancements.

38:44

Creator website

Episode guests

Eduardo Arango

Yuan Tang

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The Serving working group within Kubernetes aims to enhance model serving for AI and machine learning workloads by addressing scalability challenges.

Efforts to optimize auto-scaling and resource sharing in Kubernetes are critical for deploying large, multi-GPU models efficiently.

Deep dives

Introduction of the Serving Working Group

The formation of the Serving working group within the Kubernetes community emerged from discussions around the specific needs of AI and machine learning workloads. It addresses particular challenges faced in model serving, especially those linked to scalability and efficiency. The KSERV system has introduced advanced techniques for handling models, including pulling models from OCI images, which enhances startup times and enables capabilities like prefetching images. This working group aims to develop better foundational pieces that cater to the growing complexity of model serving and benefit the broader cloud-native ecosystem.

Intro

2min

Global Kubernetes Community Days and Insights on Cloud-Native Innovations

2min

Empowering AI Model Serving in Kubernetes

12min

Orchestration and Multi-Host Inference Challenges

13min

Engaging with Kubernetes Working Groups

6min

Collaborative Nature of Working Groups in Kubernetes

2min

Exploring Sponsorship and Development in Kubernetes Projects

2min

Yuan is a principal software engineer at Red Hat, working on OpenShift AI. Previously, he has led AI infrastructure and platform teams at various companies. He holds leadership positions in open source projects, including Argo, Kubeflow, and Kubernetes WG Serving. Yuan authored three technical books and is a regular conference speaker, technical advisor, and leader at various organizations.

Eduardo is an environmental engineer derailed into a software engineer. Eduardo has been working on making containerized environments the de facto solution for High Performance Computing(HPC) for over 8 years now. Began as a core contributor to the niche Singularity Containers, today known as Apptainer under the Linux foundation. In 2019 Eduardo moved up the ladder to work on making Kubernetes better for performance oriented applications. Nowadays Eduardo works at NVIDIA on the Core Cloud Native team working on enabling specialized accelerators into Kubernetes workloads.

Do you have something cool to share? Some questions? Let us know:

- web: kubernetespodcast.com

- mail: kubernetespodcast@google.com

- twitter: @kubernetespod

News of the week

Docker official terraform provider

Tetrate and Bloomberg Envoy AI Gateway

KubeCon+CloudNativeCon North America 2024 laptop drive

Remaining KCDs for 2024

Links from the interview

Yuan Tang

Eduardo Arango WG Serving