OpenTelemetry: What’s New with the 2nd Biggest CNCF Project?
Feb 6, 2025
auto_awesome
Morgan McLean, co-founder of OpenTelemetry and senior director at Splunk, shares insights from his extensive experience in observability for large-scale systems. He discusses the evolution of OpenTelemetry, born from OpenTracing and OpenCensus, and its crucial role in simplifying distributed tracing and application metrics. McLean anticipates that profiling will emerge as a key observability signal by 2025. He also explores the integration of AI with observability and the community's collaborative efforts aimed at improving adoption and standardization across diverse environments.
OpenTelemetry emerged to address debugging challenges in large-scale systems, combining tools from OpenTracing and OpenCensus for improved observability.
The introduction of profiling alongside existing observability signals aims to enhance application performance analysis and streamline issue resolution.
Deep dives
Foundations of OpenTelemetry
OpenTelemetry was created in response to the challenges developers faced with debugging large-scale services. Morgan McLean, one of its co-founders, recounts his early career frustrations at companies like Microsoft and Google, where existing tools lacked the capability to effectively diagnose issues during service rollouts. He and others recognized the need for standardized observability tools as the industry shifted towards virtual machines and containers, leading to a collaboration that merged several projects into OpenTelemetry. Since its launch in 2019, OpenTelemetry has become a significant contributor to the Cloud Native Computing Foundation, allowing developers to gain critical insights into application performance.
Improving Documentation and Usability
Addressing the usability challenges associated with OpenTelemetry, including its documentation and onboarding processes, is a primary focus for its ongoing development. Morgan emphasizes that the complexity of initial setup, such as configuring the OpenTelemetry Collector, has been a recurring complaint among users. To enhance accessibility, substantial progress has been made in the quality and clarity of documentation, along with efforts to simplify installation processes for various components. By streamlining setup, OpenTelemetry aims to become as user-friendly as proprietary agents, ensuring adoption across diverse environments.
The Role of Profiling in Observability
The introduction of profiling represents a significant advancement in OpenTelemetry's suite of observability tools, offering deeper insights into application performance at scale. Profiling allows developers to analyze how individual methods consume memory and CPU time, which can highlight hidden inefficiencies that impact overall system costs. Unlike existing niche profiling tools, the new functionality integrates seamlessly with existing observability signals, enabling cross-analysis of tracing, metrics, and profiling data. This capability promises to unlock a new level of performance tuning for applications, allowing developers to identify and resolve issues with unprecedented accuracy.
OpenTelemetry's Future and AI Integration
The integration of AI with OpenTelemetry is poised to shape the future of observability analytics. By leveraging OpenTelemetry’s well-defined data structure, it facilitates the application of advanced machine learning techniques to extract actionable insights from complex data sets. Morgan asserts that OpenTelemetry is uniquely positioned to support AI-driven analytics due to its standardized definitions of services and interactions, which contrasts sharply with the more chaotic nature of log-based systems. The roadmap includes ongoing enhancements to capture more data relevant to AI applications and ensure the observability framework evolves alongside industry needs.
Morgan McLean, co-founder of OpenTelemetry and senior director of product management at Splunk, has long tackled the challenges of observability in large-scale systems. In a conversation with Alex Williams onThe New Stack Makers, McLean reflected on his early frustrations debugging high-scale services and the need for better observability tools.
OpenTelemetry, formed in 2019 from OpenTracing and OpenCensus, has since become a key part of modern observability strategies. As a Cloud Native Computing Foundation (CNCF) incubating project, it’s the second most active open source project after Kubernetes, with over 1,200 developers contributing monthly. McLean highlighted OpenTelemetry’s role in solving scaling challenges, particularly in Kubernetes environments, by standardizing distributed tracing, application metrics, and data extraction.
Looking ahead, profiling is set to become the fourth major observability signal alongside logs, tracing, and metrics, with general availability expected in 2025. McLean emphasized ongoing improvements, including automation and ease of adoption, predicting even faster OpenTelemetry adoption as friction points are resolved.
Learn more from The New Stack about the latest trends in Open Telemetry: