Build Real Time Applications With Operational Simplicity Using Dozer
Data Engineering Podcast
Building a Streaming Engine and a SQL Engine in the Same Unit
The data allows us to guarantee the millisecond latency, submillisecond latency on every request. Building up a streaming engine and a SQL engine in the same unit is definitely a very complex set of engineering challenges. The value for us is being able to give the end-to-end integration of all these without worrying, without letting the user handle all these moving parts.
00:00
Transcript
Play full episode
Transcript
Episode notes
Summary
Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Despite that, it is still a complex set of capabilities. To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
- Modern data teams are using Hex to 10x their data impact. Hex combines a notebook style UI with an interactive report builder. This allows data teams to both dive deep to find insights and then share their work in an easy-to-read format to the whole org. In Hex you can use SQL, Python, R, and no-code visualization together to explore, transform, and model data. Hex also has AI built directly into the workflow to help you generate, edit, explain and document your code. The best data teams in the world such as the ones at Notion, AngelList, and Anthropic use Hex for ad hoc investigations, creating machine learning models, and building operational dashboards for the rest of their company. Hex makes it easy for data analysts and data scientists to collaborate together and produce work that has an impact. Make your data team unstoppable with Hex. Sign up today at dataengineeringpodcast.com/hex to get a 30-day free trial for your team!
- Your host is Tobias Macey and today I'm interviewing Matteo Pelati about Dozer, an open source engine that includes data ingestion, transformation, and API generation for real-time sources
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Dozer is and the story behind it?
- What was your decision process for building Dozer as open source?
- As you note in the documentation, Dozer has overlap with a number of technologies that are aimed at different use cases. What was missing from each of them and the center of their Venn diagram that prompted you to build Dozer?
- In addition to working in an interesting technological cross-section, you are also targeting a disparate group of personas. Who are you building Dozer for and what were the motivations for that vision?
- What are the different use cases that you are focused on supporting?
- What are the features of Dozer that enable engineers to address those uses, and what makes it preferable to existing alternative approaches?
- Can you describe how Dozer is implemented?
- How have the design and goals of the platform changed since you first started working on it?
- What are the architectural "-ilities" that you are trying to optimize for?
- What is involved in getting Dozer deployed and integrated into an existing application/data infrastructure?
- How can teams who are using Dozer extend/integrate with Dozer?
- What does the development/deployment workflow look like for teams who are building on top of Dozer?
- What is your governance model for Dozer and balancing the open source project against your business goals?
- What are the most interesting, innovative, or unexpected ways that you have seen Dozer used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Dozer?
- When is Dozer the wrong choice?
- What do you have planned for the future of Dozer?
Contact Info
- @pelatimtt on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Dozer
- Data Robot
- Netflix Bulldozer
- CubeJS
- JVM == Java Virtual Machine
- Flink
- Airbyte
- Fivetran
- Delta Lake
- LMDB
- Vector Database
- LLM == Large Language Model
- Rockset
- Tinybird
- Rust Language
- Materialize
- RisingWave
- DuckDB
- DataFusion
- Polars
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Hex:  Hex is a collaborative workspace for data science and analytics. A single place for teams to explore, transform, and visualize data into beautiful interactive reports. Use SQL, Python, R, no-code and AI to find and share insights across your organization. Empower everyone in an organization to make an impact with data. Sign up today at [dataengineeringpodcast.com/hex](https://www.dataengineeringpodcast.com/hex} and get 30 days free!
- Rudderstack:  Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
