Exploring Community Feedback on Profiling Formats within OpenTelemetry

OpenTelemetry is expanding beyond the traditional “three pillars of observability” and introduces a groundbreaking addition to its signals - Continuous Profiling. The new Profiling Special Interest Group (SIG) that was formed to lead the topic has already made significant advancements, to be featured at KubeCon Europe. Join us in this special panel episode of OpenObservability Talks as we explore the significance of this new dimension in understanding application behavior, optimizing performance, and gaining deeper insights into your systems. Our expert guests, Felix Geisendörfer and Ryan Perry, members of the OpenTelemetry Profiling SIG, share their insights into how Profiling enhances the OpenTelemetry framework, and update on the work for open specification and implementation. 

This special episode hosts a panel of two distinguished members of OpenTelemetry’s Profile SIG, and prominent members of the observability vendor ecosystem. Felix Geisendörfer is a Senior Staff Engineer at Datadog where he works on Continuous Profiling and contributes to the Go runtime. Before that he was working at Apple, co-founded Transloadit, contributed to node.js and inspired a generation of mad scientists to program flying robots with it. Ryan Perry is Principal Product Manager at Grafana Labs. He has built a career at various startups while actively contributing to open source projects and advancing open telemetry initiatives. Most recently he built Pyroscope, an open source continuous profiling project/company, which has been acquired by Grafana Labs.

The episode was live-streamed on 7 March 2024 and the video is available at https://www.youtube.com/watch?v=iGM67RT12gQ

OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.

We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/@openobservabilitytalks  

Show Notes:

00:00 - show intro

01:03 - episode and guests intro

04:02 - trends and advancements in the Profiling space

05:42 - from cost and performance into broader observability 

11:27 - turning profile data into metrics

12:45 - runtime vs. full host profilers and eBPF use

18:44 - pprof JFR and other existing profile standards

21:19 - profile visualizations - from flame graphs to timeline view 

22:37 - entrepreneur PoV on the profiling market

26:54 - OpenTelemetry adds profiles as a new signal

32:22 - OTel choosing a pprof extended standard  

39:06 - discrete events vs. pre-aggregated data

41:09 - use cases for processing profiling data

44:19 - OTel Profiles reference implementation 

49:11 - latest milestone and roadmap

54:44 - who’s involved in OTel Profiles

56:41 - how to follow OTel Profiles and the guests

59:34 - March community events and conferences

1:00:38 - Falco and CloudEvents projects reached CNCF graduation  

1:01:59 - Prometheus and Linkerd latest releases

1:03:29 - Netflix open-sources bpftop CLI for eBPF app performance monitoring

1:05:15 - show outro

Resources:

• Continuous Profiling: A New Observability Signal (previous episode): https://logz.io/blog/continuous-profiling-new-observability-signal-in-opentelemetry/?utm_source=devrel&utm_medium=devrel 

• OpenTelemetry extension proposal for adding Profiles: https://github.com/open-telemetry/oteps/pull/239

• OTel Profile SIG notes: https://docs.google.com/document/d/19UqPPPlGE83N37MhS93uRlxsP1_wGxQ33Qv6CDHaEp0/edit#heading=h.63a4klfdbcob

• eBPF adoption in observability - github stats: https://www.linkedin.com/feed/update/urn:li:activity:7171044354667585537/ 

• ProfilerPedia: https://profilerpedia.markhansen.co.nz/ 

• Netflix releases bpftop CLI tool: https://netflixtechblog.com/announcing-bpftop-streamlining-ebpf-performance-optimization-6a727c1ae2e5
• OpenTelemetry announces support of Profiles at KubeCon Paris 2024: https://opentelemetry.io/blog/2024/profiling/

Socials:

Twitter:⁠ https://twitter.com/OpenObserv⁠

YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠

Dotan Horovits

============

Twitter: https://twitter.com/horovits 

LinkedIn: https://www.linkedin.com/in/horovits/ 

Mastodon: @horovits@fosstodon.org

Felix Geisendörfer

===============

Twitter: https://twitter.com/felixge 

LinkedIn: https://www.linkedin.com/in/felixg2/ 

Ryan Perry

==========

Twitter: https://twitter.com/rperry_ 

Linkedin: https://www.linkedin.com/in/ryanaperry/