Scaling CI horizontally with Buildkite, Kubernetes, and multiple pipelines, with Ben Poland
Sep 30, 2025
Ben Poland, a senior staff platform engineer at Faire, dives into the transformation of CI systems from Jenkins to Buildkite. He discusses the challenges of scaling CI, addressing API throttling and optimizing workflows. Ben shares insights on splitting monolithic pipelines into service-scoped ones for better efficiency and how to manage CI across multiple Kubernetes clusters. Performance enhancements like Git mirroring and predictive provisioning are highlighted, leading to impressive results such as reduced failure rates and faster PR processing.
AI Snips
Chapters
Transcript
Episode notes
Monolith Pipeline Burned Out The Team
- Ben Poland describes inheriting a 4–5k line Groovy monolithic Jenkins pipeline that only a few platform engineers could safely change.
- That centralization caused long edit cycles, unclear failures, and frequent developer frustration blocking deployments.
Scaling Jenkins Created Massive Toil
- FAIR escalated from one Jenkins controller to over 20 to try to handle CI load, causing heavy operational toil.
- Managing many Jenkins instances forced after-hours security updates and painful migrations.
Define Clear CI Evaluation Criteria
- Evaluate CI by clear criteria: Kubernetes support, task execution needs, developer experience, observability, and security.
- Prefer proven off-the-shelf CI unless you have unique constraints that justify building one.