Ep168: Scaling Agentic Workloads: Why Reliable Infrastructure is Non-Negotiable for Enterprise AI by Anyscale

Ray originated at UC Berkeley when PhD students spent more time building clusters than ML models

Anyscale now launches 1 million clusters monthly with contributions from OpenAI, Uber, Google, Coinbase

Instacart achieved 10-100x increase in model training data using Ray's scaling capabilities

ML evolved from single-node Pandas/NumPy to distributed Spark, now Ray for multimodal data

Ray Core transforms simple Python functions into distributed tasks across massive compute clusters

Higher-level Ray libraries simplify data processing, model training, hyperparameter tuning, and model serving

Anyscale platform adds production features: auto-restart, logging, observability, and zone-aware scheduling

Unlike Spark's CPU-only approach, Ray handles both CPUs and GPUs for multimodal workloads

Ray enables LLM post-training and fine-tuning using reinforcement learning on enterprise data

Multi-agent systems can scale automatically with Ray Serve handling thousands of requests per second

Anyscale leverages AWS infrastructure while keeping customer data within their own VPCs

Ray supports EC2, EKS, and HyperPod with features like fractional GPU usage and auto-scaling