D2DO252: (Re)Building Cloudflare’s Millions-of-Logs-Per-Second Logging Pipeline

Colin Douch and Jayson Cena from Cloudflare share insights on their transition from SysLog-NG to OpenTelemetry, focusing on the need for scalability and maintainability. They discuss challenges in processing millions of logs per second while ensuring uninterrupted customer service. Their conversation covers the complexities of logging infrastructure, including performance optimization and the trade-offs between Protocol Buffers and JSON. They also touch on the importance of redundancy and team empowerment in enhancing system reliability.