More Kubernetes Than I Bargained For, with Amos Wenger
Nov 25, 2025
Amos Wenger, a developer and writer specializing in hands-on Kubernetes experiences, shares his intriguing saga of adding a home computer to his production K3s cluster. He delves into how this decision led to TLS certificate renewal failures due to NAT issues with consumer routers. The conversation highlights debugging tools like K9s and netshoot, and the unexpected IPv6 behavior encountered. Amos offers best practices for managing mixed infrastructure and encourages listeners to avoid mixing home nodes with production for a smoother experience.
AI Snips
Chapters
Transcript
Episode notes
Adding A Home Node To Production
- Amos Wenger added a Mac Studio VM to his production Kubernetes cluster to run x86 workloads he lacked in CI.
- That single home node later caused a production outage when cert renewals failed.
Cert Renewal Failure Traced To Home Node
- TLS certificate renewals started failing and Amos found cert-manager challenge pods on the home node were timing out.
- He used K9s to see pod locations and realized the home node was where failing challenges landed.
Prevent Scheduling On Incompatible Nodes
- Do taint and label non-cloud nodes to prevent critical workloads from scheduling there.
- Restrict scheduling with nodeSelectors, taints, or strict labels to avoid surprising behavior.