Realtime Data Applications Made Easier With Meroxa
Data Engineering Podcast
Kafka Connect: The Future of Data Integration
Kafka Connect just randomly creates new topics for you. There was no predictability as to how we can get that, get that stuff done. We built our own for that reason. And then just on the operational side, like, we started to use that on our for ourselves. Every, all of our service, services run on Kubernetes,. All containerizing, all that type of stuff. But look at the end of the day, right? Like, I shouldn't have to go through some weird esoteric kind of license approval process.
00:00
Transcript
Play full episode
Transcript
Episode notes
Summary
Real-time capabilities have quickly become an expectation for consumers. The complexity of providing those capabilities is still high, however, making it more difficult for small teams to compete. Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
- Your host is Tobias Macey and today I'm interviewing DeVaris Brown about the impact of real-time data on business opportunities and risk profiles
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Meroxa is and the story behind it?
- How have the focus and goals of the platform and company evolved over the past 2 years?
- Who are the target customers for Meroxa?
- What problems are they trying to solve when they come to your platform?
- Applications powered by real-time data were the exclusive domain of large and/or sophisticated tech companies for several years due to the inherent complexities involved. What are the shifts that have made them more accessible to a wider variety of teams?
- What are some of the remaining blockers for teams who want to start using real-time data?
- With the democratization of real-time data, what are the new categories of products and applications that are being unlocked?
- How are organizations thinking about the potential value that those types of apps/services can provide?
- With data flowing constantly, there are new challenges around oversight and accuracy. How does real-time data change the risk profile for applications that are consuming it?
- What are some of the technical controls that are available for organizations that are risk-averse?
- What skills do developers need to be able to effectively design, develop, and deploy real-time data applications?
- How does this differ when talking about internal vs. consumer/end-user facing applications?
- What are the most interesting, innovative, or unexpected ways that you have seen Meroxa used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Meroxa?
- When is Meroxa the wrong choice?
- What do you have planned for the future of Meroxa?
Contact Info
- @devarispbrown on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Meroxa
- Kafka
- Kafka Connect
- Conduit - golang Kafka connect replacement
- Pulsar
- Redpanda
- Flink
- Beam
- Clickhouse
- Druid
- Pinot
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Rudderstack:  RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
