MLOps.community GenAI Traffic: Why API Infrastructure Must Evolve... Again // Erica Hughberg // #296
10 snips
Mar 14, 2025 Join Erica Hughberg, Community Advocate at Tetrate, as she dives into the evolution of internet connectivity and its profound impact on AI. The conversation covers the shift from thread-based to event-driven web architectures and the transition from monolithic systems to microservices. Erica highlights how optimizing API requests with Envoy can enhance performance for large language models. She also underscores the importance of community collaboration and proactive solutions in navigating the complexities of evolving AI challenges and infrastructure.
AI Snips
Chapters
Transcript
Episode notes
The Waiter Analogy
- Erica Hughberg explains the C10k problem using a restaurant analogy.
- Thread-based proxies acted like waiters serving one table at a time, causing scaling issues as internet usage exploded.
Monoliths to Microservices
- Monolithic architectures struggled with scaling because all features resided within one large codebase.
- Microservices emerged, breaking down applications into smaller, independently scalable components.
Box Tetris
- Erica uses a "teddy bear in a box" analogy to explain microservices and Kubernetes.
- Kubernetes plays "box Tetris" to optimize resource allocation, moving services around dynamically.