Datadog, Open Telemetry, & A History of Observability
Nov 6, 2022
auto_awesome
Exploring the evolution of observability products, application telemetry collection, and microservices design patterns. Discussing the challenges and growth of observability tools, including the role of distributed tracing and market dynamics between hyperscalers and ISVs. Delving into the shift towards empowering developers, adoption of open telemetry, and the complexities in choosing between Data Dog and OpenTelemetry. Highlighting the benefits of open telemetry in enhancing observability and the impact on software developers.
Observability involves real-time system interaction via telemetry signals like tracing and logging for better understanding.
Legacy players like Splunk and DataDog have evolved to provide observability solutions through instrumenting applications with telemetry data.
The observability market is evolving rapidly with the introduction of open telemetry, enabling vendor-neutral standards and driving innovation and competition.
Deep dives
Defining Observability
Observability is the capability of a team to interact with its systems in real time and gain an understanding of those systems through telemetry signals like tracing, logging, metrics, and profiling. It entails the ability to ask unforeseen questions and obtain relevant answers. Achieving observability involves a combination of telemetry signals, a storage engine for querying them, and human interaction. It goes beyond just collecting signals and emphasizes the importance of effectively utilizing them.
Evolution of Observability
In the past, achieving observability was challenging, relying on log searches and individual host scraping. However, today, regardless of whether applications are in the cloud or on-prem, observability can be achieved by instrumenting applications to produce relevant telemetry data. Legacy players like Splunk, DataDog, Dynatrace, and AppDynamics emerged from different angles, such as logging, metrics, or APM, to develop observability solutions for modern systems. Instrumenting applications to produce telemetry and using these solutions provides observability regardless of the application's location.
The Growing Observability Market and Open Telemetry
The observability market is growing rapidly, with only a small percentage of companies effectively utilizing distributed tracing. The limitations to market growth include challenges in implementing observability efforts and the historical perception that it's an all-or-nothing approach. However, open telemetry is changing the game. Open telemetry offers the advantage of larger developer collaboration, portability, and freedom from vendor lock-in. It allows companies to coexist with legacy solutions while gradually adopting open telemetry, and it is driving innovation and competition in the observability space.
Incentives of hyperscale cloud providers
Hyperscale cloud providers like Amazon have incentives to ensure that no single player becomes dominant in the market. They benefit from the multiple tools and services offered by different ISVs and can generate revenue by charging a percentage of data sales on their marketplace. Additionally, their adoption of open telemetry allows them to integrate with various vendors without the need for custom integrations, saving resources and effort.
The rise of open telemetry and its portability
Open telemetry, a vendor-neutral standard developed by the Cloud Native Computing Foundation, enables the generation and transmission of telemetry data in a format that multiple vendors can ingest. This eliminates vendor lock-in, as users can take their portable instrumentation to any provider. Open telemetry has gained interest and contributions from major players like Google, Amazon, and Microsoft, as it allows them to ensure compatibility and enables data transmission to multiple vendors. This promotes competition and offers users the flexibility to compare and choose the best products or solutions for their needs.
Datadog, Open Telemetry, & A History of Observability
Principal Developer Advocate at Honeycomb and Former SRE Leader at Google
Liz Fong-Jones
Liz is a developer advocate at Honeycomb, a SaaS observability company that competes with Datadog, New Relic, etc. Liz previously spent over 11 years at Google Cloud as a reliability engineer and leading developer advocate across SRE, devops, and infraops.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
