Deb Roy Chowdhury, VP of Product Management at InfinyOn, shares his insights on evolving data infrastructures. He explains how legacy systems create integration chaos and how InfinyOn's Fluvio simplifies data streaming across various platforms. Chowdhury delves into Rust’s role in enhancing asynchronous tasks and the efficient handling of diverse data formats. He also discusses the challenges of no-code solutions and the importance of observability, emphasizing how a unified approach can empower data teams and streamline workflows.
The podcast emphasizes the shift from traditional healthcare roles to data-centric positions, highlighting the growing importance of data analysis in improving patient care.
It discusses the challenges posed by legacy systems in data analytics, advocating for modern big data technologies to enhance performance and efficiency.
The use of Rust in building the Fluvio data streaming platform showcases a trend towards adopting efficient solutions for complex data workflows and integration challenges.
Deep dives
Career Transition and Data's Role in Healthcare
The discussion highlights the transition from traditional healthcare roles to roles centered around data analysis. As healthcare systems became digitized, there was a growing need to analyze population health and improve quality of care, leading to more data-centric roles in the field. Early experiences involved descriptive statistics and the handling of previously paper-based systems, showcasing a paradigm shift towards data accumulation and analysis. This transition underscores the importance of data management in modern healthcare, ultimately shaping approaches to patient care and operational efficiency.
Challenges of Legacy Systems
The complexities of legacy systems in data analytics are examined, particularly with large databases and slow data processing speeds. Experiences at a previous workplace revealed the difficulties of managing and optimizing SQL queries across thousands of tables, making data retrieval time-consuming and inefficient. Efforts to improve the system involved the transition from traditional Microsoft-based environments to modern big data technologies such as Hadoop and Spark, which, while promising, presented their own challenges of complexity and performance issues. The struggle for optimization is a common theme as teams work to eliminate bottlenecks in reporting and information retrieval.
The Emergence of Rust in Data Engineering
Rust's growing prominence in data engineering is explored, particularly as a solution for building efficient streaming data processes. The speaker's experience with data aggregation for e-commerce highlighted the need for an optimal solution that could reduce costs associated with managing multiple tools. The integration of Rust within the development of a new data processing system, Fluvio, was driven by its performance capabilities and the need for a modular architecture that minimizes dependency on multiple systems. This marks a significant shift towards adopting Rust for its potential to streamline complex data workflows.
Fluvio and Modular Data Infrastructure
Fluvio is positioned as a unified system designed to streamline various data processing tasks while minimizing the need for multiple tools. The goal is to create a comprehensive solution that provides modular and composable infrastructure for data integration, processing, and visualization. This innovative approach aims to consolidate functionalities typically spread across separate systems, thus enhancing efficiency and reducing operational costs. The focus on stateful data flows and real-time processing showcases an ambition to meet contemporary data demands effectively.
Community and Future of Rust in Data
The speaker emphasizes the importance of community engagement in the Rust ecosystem and the potential for collaboration on projects within the data space. With a commitment to continuous improvement, they express a desire to work alongside other Rust developers to shape future data solutions. Observability and integration capabilities are highlighted as central to Fluvio’s philosophy, ensuring robust monitoring and troubleshooting while remaining adaptable to user needs. This forward-looking perspective reflects both an appreciation for Rust's strengths and a clear vision for innovation in data processing.
Picture this: Your organization's data infrastructure resembles a busy kitchen with too many cooks. You're juggling Kafka for messaging, Flink for processing, Spark for analytics, Airflow for orchestration, and various Lambda functions scattered about. Each tool excellent at its job, but together they've created a complex feast of integration challenges. Your data teams are spending more time managing tools than extracting value from data. InfinyOn reimagines this chaos with a radically simple approach: a unified system for data streaming that runs everywhere. Unlike traditional solutions that struggle at the edge, InfinyOn gracefully handles data streams from IoT devices to cloud servers. And instead of cobbling together different tools, developers can build complete data pipelines using their preferred languages - be it Rust, Python, or SQL - with built-in state management.
At the heart of InfinyOn is Fluvio, a Rust-based data streaming platform that's fast, reliable, and easy to use.
About InfinyOn
Data pipelines are often slow, unreliable, and complex. InfinyOn, the creators of Fluvio, aims to fix this. Built in Rust, Fluvio offers fast, reliable data streaming. It lets you build event-driven pipelines quickly, running as a single 37 MB binary. With features like SmartModules, it handles various data types efficiently. Designed for developers, it offers a clean API and intuitive CLI. Streamline your data infrastructure at infinyon.com/rustinprod.
About Deb Roy Chowdhury
For fifteen years, Deb has been a behavioral detective, piecing together human decision-making through conversations, data, and research. His passion lies in product innovation—finding that sweet spot where desirability, viability, and feasibility converge. From 7-person startups to tech giants of 165,000, he helped build products that people love. Deb is currently the VP of Product Management at InfinyOn, where he leads the product strategy and roadmap for Fluvio, a Rust-based data streaming platform.
Links From The Show
Polars - Fast DataFrame library implemented in Rust
Apache Arrow - Cross-language development platform for in-memory data
Arroyo - SQL-based data streaming platform in Rust
"Rust in Production" is a podcast by corrode, a company that helps teams adopt Rust. We offer training, consulting, and development services to help you succeed with Rust. If you want to learn more about how we can help you, please get in touch.
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode