Building The DataDog Platform For Processing Timeseries Data At Massive Scale
Data Engineering Podcast
Custom File Formats for Onboarding Engineers?
The goal of those custom file formats is to allow customers to see data as soon as possible. And we try to make sure that within a minute you can get data. For analytics purposes and for other more flexible use cases, we figured out that we actually want to store in Parquet. But essentially like our system is optimized. It doesn't matter if it's like Parquet or some other file format as long as you get data with a certain Schema. You don't really care as long as that schema is the same.
Summary
DataDog is one of the most successful companies in the space of metrics and monitoring for servers and cloud infrastructure. In order to support their customers, they need to capture, process, and analyze massive amounts of timeseries data with a high degree of uptime and reliability. Vadim Semenov works on their data engineering team and joins the podcast in this episode to discuss the challenges that he works through, the systems that DataDog has built to power their business, and how their teams are organized to allow for rapid growth and massive scale. Getting an inside look at the companies behind the services we use is always useful, and this conversation was no exception.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
- You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
- Your host is Tobias Macey and today I’m interviewing Vadim Semenov about how data engineers work at DataDog
Interview
- Introduction
- How did you get involved in the area of data management?
- For anyone who isn’t familiar with DataDog, can you start by describing the types and volumes of data that you’re dealing with?
- What are the main components of your platform for managing that information?
- How are the data teams at DataDog organized and what are your primary responsibilities in the organization?
- What are some of the complexities and challenges that you face in your work as a result of the volume of data that you are processing?
- What are some of the strategies which have proven to be most useful in overcoming those challenges?
- Who are the main consumers of your work and how do you build in feedback cycles to ensure that their needs are being met?
- Given that the majority of the data being ingested by DataDog is timeseries, what are your lifecycle and retention policies for that information?
- Most of the data that you are working with is customer generated from your deployed agents and API integrations. How do you manage cleanliness and schema enforcement for the events as they are being delivered?
- What are some of the upcoming projects that you have planned for the upcoming months and years?
- What are some of the technologies, patterns, or practices that you are hoping to adopt?
Contact Info
- @databuryat on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Links
- DataDog
- Hadoop
- Hive
- Yarn
- Chef
- SRE == Site Reliability Engineer
- Application Performance Management (APM)
- Apache Kafka
- RocksDB
- Cassandra
- Apache Parquet data serialization format
- SLA == Service Level Agreement
- WatchDog
- Apache Spark
- Apache Pig
- Databricks
- JVM == Java Virtual Machine
- Kubernetes
- SSIS (SQL Server Integration Services)
- Pentaho
- JasperSoft
- Apache Airflow
- Apache NiFi
- Luigi
- Dagster
- Prefect
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
The AI-powered Podcast Player
