KubeFM The Data Engineer's guide to optimizing Kubernetes, with Niels Claeys
Oct 14, 2025
Niels Claeys, a lead engineer at Dataminded and expert in Kubernetes optimization, shares insights on building Conveyor, a data platform processing over 1.5 million core hours monthly. He reveals how switching scheduler strategies can cut costs significantly while enhancing resource use. Niels also discusses achieving 97% spot instance utilization and the importance of multi-type diversification. He emphasizes the need for simplicity in coding and effective communication in tech, alongside practical tips for scaling and optimizing workloads.
AI Snips
Chapters
Transcript
Episode notes
Operating Scale In Core Hours
- Conveyor handles between 500,000 and 1.5 million core hours monthly across customers.
- That translates roughly to 200–300 standard nodes running continuously if translated to steady state.
Node Overhead Reduces Usable Capacity
- Kubernetes node resources are reduced by OS reservations and daemonsets, lowering available memory and CPU for jobs.
- Niels notes a 16 GiB/4 vCPU node effectively offers about 14 GiB to workloads after overhead.
Bin Pack By Preferring Most-Allocated
- Switch scheduler strategy from least-allocated to most-allocated to consolidate pods and enable faster scale-down.
- This change typically yields immediate 10–15% cost savings without user changes.