The New Stack Podcast

How Oracle Is Meeting the Infrastructure Needs of AI

Jan 23, 2025

Sudha Raghavan, SVP for Developer Platform at Oracle Cloud Infrastructure, discusses the seismic shift in infrastructure needs driven by AI's rapid adoption. She highlights the explosive demand for GPUs and the challenges posed by continuous workloads like large language model training. Raghavan elaborates on Oracle's innovative GPU superclusters and improvements in Kubernetes for better job management and observability. The conversation also touches on the evolution of AI pipelines and the critical trade-offs in AI model selection to meet specific business demands.

27:28

Creator website

Episode guests

Sudha Raghavan

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The rapid adoption of generative AI has drastically increased demand for GPUs, necessitating innovative infrastructure solutions like Oracle's massive GPU superclusters.

To effectively manage AI workloads, Oracle is enhancing Kubernetes functionalities to support stateful operations and improve job scheduling for GPU resources.

Deep dives

The Impact of GPU Demand on Infrastructure

The demand for GPUs has surged rapidly due to advancements in generative AI, leading to new infrastructure requirements. Unlike traditional workloads that experience peaks and troughs, GPUs operate at peak levels consistently, demanding substantial power and posing a risk of hardware failure. Oracle Cloud Infrastructure (OCI) is addressing these challenges by developing a massive GPU supercluster, consisting of over 131,000 nodes, to handle extensive training jobs for its clients. This shift emphasizes the need for close physical placement of nodes to minimize latency and ensure efficient state management across large batch jobs.

Intro

2min

Navigating GPU Management in AI

14min

Navigating AI/ML Pipelines with Kubeflow and MLOps

3min

Navigating DevOps in AI Workloads

6min

Navigating Trade-Offs in AI Model Selection

2min

Generative AI is a data-driven story with significant infrastructure and operational implications, particularly around the rising demand for GPUs, which are better suited for AI workloads than CPUs. In an episode ofThe New Stack Makersrecorded at KubeCon + CloudNativeCon North America, Sudha Raghavan, SVP for Developer Platform at Oracle Cloud Infrastructure, discussed how AI’s rapid adoption has reshaped infrastructure needs.

The release of ChatGPT triggered a surge in GPU demand, with organizations requiring GPUs for tasks ranging from testing workloads to training large language models across massive GPU clusters. These workloads run continuously at peak power, posing challenges such as high hardware failure rates and energy consumption.

Oracle is addressing these issues by building GPU superclusters and enhancing Kubernetes functionality. Tools like Oracle’s Node Manager simplify interactions between Kubernetes and GPUs, providing tailored observability while maintaining Kubernetes’ user-friendly experience. Raghavan emphasized the importance of stateful job management and infrastructure innovations to meet the demands of modern AI workloads.

Learn more from The New Stack about how Oracle is addressing the GPU demand for AI workloads with its GPU superclusters and enhancing Kubernetes functionality:

Oracle Code Assist, Java-Optimized, Now in Beta

Oracle’s Code Assist: Fashionably Late to the GenAI Party

Oracle Unveils Java 23: Simplicity Meets Enterprise Power

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.