Scaling CI horizontally with Buildkite, Kubernetes, and multiple pipelines, with Ben Poland

Ben Poland walks through Faire's complete CI transformation, from a single Jenkins instance struggling with thousands of lines of Groovy to a distributed Buildkite system running across multiple Kubernetes clusters.

He details the technical challenges of running CI workloads at scale, including API rate limiting, etcd pressure points, and the trade-offs of splitting monolithic pipelines into service-scoped ones.

You will learn:

• How to architect CI systems that match team ownership and eliminate shared failure points across services

• Kubernetes scaling patterns for CI workloads, including multi-cluster strategies, predictive node provisioning, and handling API throttling

• Performance optimization techniques like Git mirroring, node-level caching, and spot instance management for variable CI demands

• Migration strategies and lessons learned from moving away from monolithic CI, including proof-of-concept approaches and avoiding the sunk cost fallacy

Sponsor

This episode is brought to you by Testkube—where teams run millions of performance tests in real Kubernetes infrastructure. From air-gapped environments to massive scale deployments, orchestrate every testing tool in one platform. Check it out at testkube.io

More info

• Find all the links and info for this episode here: https://ku.bz/klBmzMY5-

• Interested in sponsoring an episode? Learn more.