Navigating Data Observability and SLOs

Supported by Our Partners

• Sonar —  Trust your developers – verify your AI-generated code.

• Vanta —Automate compliance and simplify security with Vanta.

—

In today's episode of The Pragmatic Engineer, I'm joined by Charity Majors, a well-known observability expert – as well as someone with strong and grounded opinions. Charity is the co-author of "Observability Engineering" and brings extensive experience as an operations and database engineer and an engineering manager. She is the cofounder and CTO of observability scaleup Honeycomb.

Our conversation explores the ever-changing world of observability, covering these topics:

• What is observability? Charity’s take

• What is “Observability 2.0?”

• Why Charity is a fan of platform teams

• Why DevOps is an overloaded term: and probably no longer relevant

• What is cardinality? And why does it impact the cost of observability so much?

• How OpenTelemetry solves for vendor lock-in 

• Why Honeycomb wrote its own database

• Why having good observability should be a prerequisite to adding AI code or using AI agents

• And more!

—

Timestamps

(00:00) Intro 

(04:20) Charity’s inspiration for writing Observability Engineering

(08:20) An overview of Scuba at Facebook

(09:16) A software engineer’s definition of observability 

(13:15) Observability basics

(15:10) The three pillars model

(17:09) Observability 2.0 and the shift to unified storage

(22:50) Who owns observability and the advantage of platform teams 

(25:05) Why DevOps is becoming unnecessary 

(27:01) The difficulty of observability 

(29:01) Why observability is so expensive 

(30:49) An explanation of cardinality and its impact on cost

(34:26) How to manage cost with tools that use structured data 

(38:35) The common worry of vendor lock-in

(40:01) An explanation of OpenTelemetry

(43:45) What developers get wrong about observability 

(45:40) A case for using SLOs and how they help you avoid micromanagement 

(48:25) Why Honeycomb had to write their database 

(51:56) Companies who have thrived despite ignoring conventional wisdom

(53:35) Observability and AI 

(59:20) Vendors vs. open source

(1:00:45) What metrics are good for 

(1:02:31) RUM (Real User Monitoring) 

(1:03:40) The challenges of mobile observability 

(1:05:51) When to implement observability at your startup 

(1:07:49) Rapid fire round

—

The Pragmatic Engineer deepdives relevant for this episode:

• How Uber Built its Observability Platform https://newsletter.pragmaticengineer.com/p/how-uber-built-its-observability-platform 

• Building an Observability Startup https://newsletter.pragmaticengineer.com/p/chronosphere 

• How to debug large distributed systems https://newsletter.pragmaticengineer.com/p/antithesis 

• Shipping to production https://newsletter.pragmaticengineer.com/p/shipping-to-production 

—

See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠

—

Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.