A 3-node Kubernetes cluster with Prometheus will ship around 40k active series by default! Do we really need all that data? The current state of Kubernetes open source monitoring is in need of improvement. High churn rate of pod metrics, proliferation of metrics with low usage, and configuration complexity are some of the issues that need to be addressed.

I discussed this topic with Aliaksandr Valialkin, CTO at VictoriaMetrics and creator of the open source project. We discussed the common problems, as well as directions and best practices to overcome some of these complexities as individuals and as a community. We also discussed VictoriaMetrics open source project and how it addresses some of these challenges.

Aliaksandr a Golang engineer, who likes writing simple and performant code and creating easy-to-use programs. Sometimes these hard-to-match requirements work together, like in the VictoriaMetrics case.

The episode was live-streamed on 24 January 2023 and the video is available at https://www.youtube.com/live/Z-58C8HFGb8

OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.

We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.
https://www.twitch.tv/openobservability
https://www.youtube.com/@openobservabilitytalks

Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/

Show Notes:

monitoring microservice system, app and communications
high churn rate for pod metrics
Kubernetes produces too many metrics by defaults, most of which are unused
recommended listing of metrics
removing unused metric labels to reduce cardinality
Prometheus native (exponential buckets) historgrams
Configuration complexity with multiple deployments
OpenTelemetry and OpenMetrics open specifications
collecting system metrics and application metrics uniformly
VictoriaMetrics essentials
VictoriaMetrics extensions beyond Prometheus
a full stack monitoring collection, analysis and alerting
how to join the VictoriaMetrics community
industry update: 2023 cloud native predictions post by CNCF CTO

Resources:

Why Prometheus cannot query remote storage in an expected way via remote_read protocol - https://github.com/prometheus/prometheus/issues/4456
VictoriaMetrics scaling to 100 million metrics per second https://www.youtube.com/watch?v=xfed9_Q0_qU
https://victoriametrics.com/
https://github.com/VictoriaMetrics/VictoriaMetrics
https://docs.victoriametrics.com/#community-and-contributions

Socials:

Twitter: https://twitter.com/OpenObserv
Twitch: https://www.twitch.tv/openobservability
YouTube: https://www.youtube.com/@openobservabilitytalks

Dotan Horovits

Twitter: https://twitter.com/horovits
LinkedIn: https://www.linkedin.com/in/horovits/

Aliaksandr Valialkin

Twitter: https://twitter.com/valyala
LinkedIn: https://www.linkedin.com/in/valyala/

Ruby Upgrades & RAM Shortages

Remote Ruby

When to enable YJIT for Rails

The AI-powered Podcast Player