OpenObservability Talks

A 3-node Kubernetes cluster with Prometheus will ship around 40k active series by default! Do we really need all that data? The current state of Kubernetes open source monitoring is in need of improvement. High churn rate of pod metrics, proliferation of metrics with low usage, and configuration complexity are some of the issues that need to be addressed. I discussed this topic with Aliaksandr Valialkin, CTO at VictoriaMetrics and creator of the open source project. We discussed the common problems, as well as directions and best practices to overcome some of these complexities as individuals and as a community. We also discussed VictoriaMetrics open source project and how it addresses some of these challenges. Aliaksandr a Golang engineer, who likes writing simple and performant code and creating easy-to-use programs. Sometimes these hard-to-match requirements work together, like in the VictoriaMetrics case. The episode was live-streamed on 24 January 2023 and the video is available at https://www.youtube.com/live/Z-58C8HFGb8 OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/@openobservabilitytalks Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: monitoring microservice system, app and communications high churn rate for pod metrics Kubernetes produces too many metrics by defaults, most of which are unused recommended listing of metrics removing unused metric labels to reduce cardinality Prometheus native (exponential buckets) historgrams Configuration complexity with multiple deployments OpenTelemetry and OpenMetrics open specifications collecting system metrics and application metrics uniformly VictoriaMetrics essentials VictoriaMetrics extensions beyond Prometheus a full stack monitoring collection, analysis and alerting how to join the VictoriaMetrics community industry update: 2023 cloud native predictions post by CNCF CTO Resources: Why Prometheus cannot query remote storage in an expected way via remote_read protocol - https://github.com/prometheus/prometheus/issues/4456  VictoriaMetrics scaling to 100 million metrics per second https://www.youtube.com/watch?v=xfed9_Q0_qU https://victoriametrics.com/ https://github.com/VictoriaMetrics/VictoriaMetrics https://docs.victoriametrics.com/#community-and-contributions Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/@openobservabilitytalks Dotan Horovits  Twitter: https://twitter.com/horovits  LinkedIn: https://www.linkedin.com/in/horovits/   Aliaksandr Valialkin Twitter: https://twitter.com/valyala LinkedIn: https://www.linkedin.com/in/valyala/

So much has been going on with the Prometheus project and its ecosystem, that it’s time to have a proper catch up. And there’s no better person to walk us through it than Julien Pivotto, who debriefed the community last month at PromCon. Julien Pivotto is a maintainer of Prometheus, the open source monitoring and alerting solution. He is the co-founder of the company O11y, that provides premium support for open source observability tools such as Prometheus, Thanos and Grafana. The episode was live-streamed on 19 December 2022 and the video is available at https://www.youtube.com/watch?v=Vui4EgveUxg OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/@openobservabilitytalks Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: Prometheus is 10 years old Prometheus mission statement and directions New Agent Mode for work with external backends Service Discovery ecosystem, plugins and updates Time series database basics and updates New support for native histograms Examplars in Prometheus to correlate metrics to traces PromQL query language updates PromLens contribution to Prometheus Prometheus UI updates Visualization options: Grafana and Perses  Alertmanager updates Windows exporter, MySQL and other new exporters Long term support for Prometheus project Thanos, Cortex, Mimir - ecosystem update Prometheus community   Resources: PromCon EU 2022 talk: https://www.youtube.com/watch?v=2Wlza5jrS-U Prometheus GitHub: https://github.com/prometheus/prometheus Why Your Monitoring Dashboard May Be Lying to You: https://horovits.medium.com/ca477e80589e Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Host: https://twitter.com/horovits 

At Meta (Facebook, Instragram et al) everything is data, and data driven approach is the rule, from product to engineering, from HR to finance. This is also how the team at Meta treats observability. Let’s see how we treat observability as a data analytics problem, and what you can implement, even if you’re not a hyperscaler. On this episode I’ll host David Ostrovsky from Meta. David is a software developer with over 20 years of industry experience, speaker, trainer, blogger and co-author of “Pro Couchbase Server”. He specializes in large-scale distributed system architecture. The episode was live-streamed on 7 November 2022 and the video is available at https://youtu.be/1l0HKUDoX4Q OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: Addressing business observability Machine learning and predictions in observability From the business inwards, using SLOs Accelerate engineering quality with developer observability Organizational and communications aspects of high scale observability Actionable observability How small-medium size orgs can achieve a similar effect OpenTelemetry demo is GA PromLens is open sourced and contributed to Prometheus Resources: TEMPLE signals for observability: https://medium.com/@YuriShkuro/temple-six-pillars-of-observability-4ac3e3deb402 PromLens open sourced: https://promlabs.com/blog/2022/10/25/promlabs-and-chronosphere-open-source-the-promlens-query-builder Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg

Everyone’s talking about Platform Engineering these days. Even Gartner featured it in its Hype Cycle for Software Engineering 2022. But what is Platform Engineering really about? Is it the next stage in the evolution of DevOps? Is it just a fancy rebrand for DevOps or SRE? And how does observability pertain to platform engineering? On this episode of OpenObservability Talks Horovits hosted George Hantzaras, Director of Cloud Platform Engineering at Citrix. George is a distributed systems expert and a hands-on engineering leader with focus on delivering B2B cloud services at scale. Coming from a DevOps background, he focuses on implementing SRE at enterprise scale working with cloud native technologies. He has been organizing the Athens Cloud Computing Meetup since 2016 and the Athens Hashicorp User Group. Most recently, he has been a speaker at global events like Hashiconf, DeveloperWeek, Voxxed Days, DevNexus and more, focusing on reliability engineering, agile leadership, scaling engineering teams, and entrepreneurship. The episode was live-streamed on 6 October 2022 and the video is available at https://youtu.be/zJGMUVY6fDM OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: Why Platform Engineering? Different teams employ different stacks. How does Platform Engineering fits in? Platform Engineering at Citrix How does it map to the common DevOps maturity models? Is Platform Engineering different from PaaS? Platform as a product Culture element is central in Platform Engineering Generating buy-in within the product’s engineering Metrics and quantifying the benefits of Platform Engineering Balancing flexibility and simplicity, and striking the right level of abstraction Delimitation between what’s developed by Platform vs. Product teams. Generating Golden Paths for engineering Observability and Platform Engineering Resources: George Hantzaras presentations Designing Golden Paths platformengineering.org Platform Engineering KPIs Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg

Instrumentation is that black magic that makes our application emit traces, logs, metrics or other telemetry. How does it work? What options are available in different programming languages, such as Java, Python and Go? What does OpenTelemetry offer in this domain?   On this episode of OpenObservability Talks I hosted Eden Federman, Co-Founder & CTO at keyval, a company focused on making observability simpler. Eden is the creator of two open source projects: Odigos and Go automatic instrumentation (now part of OpenTelemetry). Eden is passionate about everything related to observability and performance monitoring. He also created kubectl-flame, a profiler for Kubernetes. The episode was live-streamed on 22 September 2022 and the video is available at https://youtu.be/VFykWV1mLAI OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: What is instrumentation Manual and Automatic instrumentation Different languages offer different options Java instrumentation capabilities Go instrumentation capabilities Instrumentation when using programming frameworks eBPF use in auto-instrumentation New OpenTelemetry SIG for Go auto-instrumentation Odigos open source project Best practices for instrumentation Resources: Instrumentation basics and OpenTelemetry support eBPF Automatic instrumentation for Go Odigos (Observability Control Plane) OpenTelemetry Go auto instrumentation SIG How OpenTelemetry works under the hood in JavaScript Spring Boot instrumentation Spring Cloud Sleuth 1.1.0 released Socials: Twitter:⁠ https://twitter.com/OpenObserv⁠ YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠

In this episode, hosted by Jonah Kowall we are excited to introduce Pranay Prateek the co-founder and creator of SigNoz to the audience. He is an entrepreneur who has been building an interesting and exciting open-source observability platform. The goal is a unified interface and data storage system which is scalable and manageable much more easily than today’s disjointed open-source tools. SigNoz has a goal of unifying tracing, metrics, and logging in a single platform and interface. We’ll dig into the evolution of this unique project from its creation until today, along with the adoption of the technology and contribution from the community. We will discuss the data architecture and their move from Druid towards ClickHouse, which is always a fascinating topic. Finally, we’ll elaborate on the future of SigNoz and where things are going with the project. Join me, Jonah Kowall, as I speak with Pranay Prateek on this episode of OpenObservability Talks. The podcast episodes are available for listening on your favorite podcast app and on this YouTube channel. Socials: Twitter:⁠ https://twitter.com/OpenObserv⁠ YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠

We all collect logs, metrics and perhaps traces and other data types, in support of our observability. But this can get expensive pretty quickly, especially in microservices based systems, in what is commonly known as “the cardinality problem”.   On this episode of OpenObservability Talks I’ll host Ben Sigelman, co-founder and the GM of Lightstep, to discuss this data problem and how to overcome it. Ben architected Google’s own planet-scale metrics and distributed tracing systems (still in production today), and went on to co-create the open-source OpenTracing and OpenTelemetry projects, both part of the CNCF. The episode was live-streamed on 12 July 2022 and the video is available at https://youtu.be/gJhzwP-mZ2k OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and pitch in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: The difference between monitoring, observability and APM What comprises the cost of observability How common is the knowledge of cardinality and how to add metrics Controlling cost with sampling, verbosity and retention Lessons from Google’s metrics and tracing systems Using metric rollups and aggregations intelligently Semantic conventions for logs, metrics and traces OpenCost project New research paper by Meta on schema-first approach to application telemetry metadata OTEL code contributions - published stats Resources: Monitoring vs. observability: https://twitter.com/el_bhs/status/1349406398388400128 The two drivers of cardinality: https://twitter.com/el_bhs/status/1360276734344450050 Sampling vs verbosity: https://twitter.com/el_bhs/status/1440750741384089608 Observing resources and transactions: https://twitter.com/el_bhs/status/1372636288021524482 Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg

OpenTelemetry is one of the most fascinating and ambitious open source projects of this era. It’s currently the second most active project in the CNCF (the Cloud Native Computing Foundation), with only Kubernetes being more active. The entire industry is aligning behind this project, including incumbent monitoring vendors that were deeply vested in proprietary and closed-source agents to that end. In this episode of OpenObservability Talks I’ll host Alolita Sharma to discuss OpenTelemetry, its origins and mission statement, as well as updates hot off the press from the recent KubeCon conference in Valencia about releases and future plans. Alolita is co-chair of the CNCF Technical Advisory Group for Observability, member of the OpenTelemetry Governance Committee and a board director of the Unicode Consortium. She has served on the boards of the OSI and SFLC.in. Alolita has led engineering teams at Wikipedia, Twitter, PayPal, IBM and AWS. Two decades of doing open source continue to inspire her. The episode was live-streamed on 15 June 2022 and the video is available at https://youtu.be/IK2TWOzDUBI  OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and pitch in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg You can read the recap post: https://logz.io/blog/opentelemetry-roadmap-and-latest-updates/?utm_source=devrel&utm_medium=devrel Show Notes: Hot updates from KubeCon EMEA 2022 Alolita Sharma introduction The state of OpenTelemetry When OpenTelemetry Logging is expecting GA The onboarding challenge of instrumentation Client side instrumentation and real user monitoring Adding continuous profiling telemetry to OpenTelemetry Interoperability between OpenTelemetry and Prometheus Challenges in OpenTelemetry and observability Where OpenTelemetry is heading next Jaeger OSS now accept OTLP (OpenTelemetry protocol) Resources: OpenTelemetry Metrics reaches RC: https://opentelemetry.io/blog/2022/metrics-announcement/ OpenTelemetry guide: https://logz.io/learn/opentelemetry-guide/ CI/CD Observability: https://horovits.medium.com/fighting-slow-and-flaky-ci-cd-pipelines-starts-with-observability-19da2ac94677 Jaeger can now accept OpenTelemetry protocol https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c OTel Community Day summary: http://paulsbruce.io/blog/2022/06/opentelemetry-community-day-austin-2022 Contextual Logging in Kubernetes 1.24 https://kubernetes.io/blog/2022/05/25/contextual-logging/  PolarSignals announced FrostDB https://www.polarsignals.com/blog/posts/2022/05/04/introducing-arcticdb/  Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Dotan Horovits ============ Twitter: @horovits LinkedIn: in/horovits Mastodon: @horovits@fosstodon Alolita Sharma ============ Twitter: @alolita LinkedIn: https://www.linkedin.com/in/alolita/

Developers hate monitoring, but we need it. We need it in many points of the software development lifecycle: before deprecating an API, before launching a new feature, after launching the feature, and more. In fact, monitoring needs can vary much more than the classic Ops monitoring. In this episode I’ll host Liran Haimovitch to discuss how to determine what developers should be monitoring, the difference between observability for Dev and for Ops, and how observability fits into our current dev tools, dev stack and dev processes. Liran is the Co-Founder and CTO of Rookout. He’s an Observability and Instrumentation expert with a deep understanding of Java, Python, Node, and C++. Liran has broad experience in cybersecurity and compliance from his past roles. When not coding, you can find Liran hosting his podcast, speaking at conferences, writing about his tech adventures, and trying out the local cuisine when traveling. The episode was live-streamed on 10 May 2022 and the video is available at https://youtu.be/OaHQp-qnVN0  OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and pitch in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: Which data do we need to collect for our observability How is observability for dev different from ops How does observability fit into dev tool stack Snapshots provide deep-dive telemetry signal Dynamic instrumentation Snapshots support in programming languages and runtimes Open source standardization around snapshots The cost associated with observability Google is applying to contribute Istio to the CNCF Shopify case study for observability team Resources: Istio applying to the CNCF: https://istio.io/latest/blog/2022/istio-has-applied-to-join-the-cncf/ Shopify case study for Observability team: https://ericmustin.substack.com/p/notes-on-an-observability-team?s=r Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg

OpenSearch is a community-driven, open-source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. The OpenSearch project started just over a year ago and is now the open-source alternative to ELK, which is no longer open source. The team has spent much of the last year getting the project going, but there was innovation as well. We will cover and discuss what OpenSearch has accomplished, but more importantly what’s coming next, including a big 2.0 release. We are joined in this episode by Eli Fisher, who is the product lead at AWS, working on the OpenSearch project. He’ll dive into recent launches, including several observability features, and innovations planned for 2.0 and beyond.    The podcast episodes are available for listening on your favorite podcast app and on this YouTube channel.   We live-stream the episodes, and you’re welcome to join the stream here on YouTube Live or at https://www.twitch.tv/openobservability​.