Data Engineering Podcast

Chakravarti Kotaru, Director of a data platform at a leading online travel company with nearly two decades in scalable architectures, shares his expertise on data operations. He discusses the evolution from DevOps to platform engineering, emphasizing centralized management and automation with AWS Service Catalog. Kotaru elaborates on the challenges of database migrations, the integration of AI and ML for enhanced efficiency, and the vital role of organizational buy-in in successful data initiatives. A fascinating glimpse into modern data platform strategies!

Shinji Kim, Founder and CEO of SelectStar, shares insights on the evolving role of semantic layers in AI. He discusses the journey from statistical analysis to data governance, highlighting challenges enterprises face with data access. The conversation covers the shift from centralized to decentralized data teams and the importance of metadata management. Shinji emphasizes the critical role of semantic modeling for business intelligence and how AI can enhance data accuracy. He also explores the future of semantic modeling in data warehouses, addressing operationalization challenges.

Tulika Bhatt, a senior software engineer at Netflix specializing in impression data, shares her journey from BlackRock and Verizon to shaping data services at a top streaming service. She discusses the challenges of balancing off-the-shelf solutions with custom systems, utilizing technologies like Spark and Flink. Tulika dives into the intricacies of ensuring data quality and observability, emphasizing automation and robust alerting strategies. She also explores the integration of AI in data engineering, highlighting its potential and the hurdles faced in maximizing efficiency.

Sida Shen, a product manager at CelerData and a contributor to StarRocks, dives into the innovative world of high-performance analytical databases. He shares the origins of StarRocks, illustrating its evolution from Apache Doris into a robust Lakehouse query engine. Topics include handling high concurrency and low latency queries, bridging traditional OLAP with lakehouse architecture, and the importance of integration with formats like Apache Iceberg. Sida also emphasizes the challenges of denormalization and real-time data processing in modern analytics.

Derek Collison, the creator of NATS and CEO of Synadia, shares insights from his impressive background at Google and VMware. He discusses how NATS revolutionizes messaging systems with innovative features like the circuit breaker pattern and Jetstream. Derek highlights NATS’s advantages in edge computing, emphasizing its resilience and data persistence capabilities. He also addresses the challenges of open-source technology and shares thoughts on the future of connectivity in modern distributed systems, proving NATS's versatility across various industries.

Victor Kessler, co-founder of Vakama and developer of Lakekeeper, dives into the world of advanced lakehouse management with a focus on Apache Iceberg. He discusses the pivotal role of metadata in data actionability and the evolution of data catalogs. Victor highlights innovative features of Lakekeeper, like integration with OpenFGA for access control and its deployment using Rust on Kubernetes. He also addresses the challenges of migrating data catalogs and the importance of community involvement in open-source projects for better data management.

In this engaging conversation, Jeremy Edberg, CEO of DBOS and former tech lead at companies like Reddit and Netflix, discusses the vital concept of durable execution in data systems. He reveals how DBOS's serverless platform enhances local resilience and simplifies the intricacies of data pipelines. Jeremy emphasizes the significance of version management for long-running workflows and introduces the Transact library, which boosts reliability and efficiency. This episode is a treasure trove for anyone interested in optimizing data workflows and reducing operational headaches.

Roman Gershman, CTO and founder of Dragonfly DB, shares his journey from Google to creating a high-speed alternative to Redis. He dives into the challenges of developing in-memory databases, focusing on performance, scalability, and cost efficiency. Roman discusses operational complexities users face, while highlighting Dragonfly's compatibility with Redis and innovations like SSD tiering. He also explores programming trade-offs between C++ and Rust, emphasizing adaptability in database development and the importance of community feedback in shaping future advancements.

Sean Knapp, Founder and CEO of Ascend.io, shares his expertise in data engineering and AI's transformative role. He discusses how AI can streamline workflows, alleviate burdens for data engineers, and enhance productivity by automating tasks. Sean highlights challenges like data governance and the integration of AI into existing systems. The conversation also touches on bridging the gap between junior and senior engineers using AI as a collaborative tool, as well as the future potential of AI to revolutionize data engineering processes.

Pete DeJoy, co-founder and product lead at Astronomer, shares his extensive experience with Airflow, discussing its evolution and upcoming enhancements in Airflow 3. He highlights Astronomer's commitment to improving data operations and community involvement. The conversation dives into the critical role of data observability through Astra Observe, innovative use cases like the Texas Rangers in-game analytics, and the shifting landscape of data engineering roles, emphasizing collaboration and advanced tooling in the modern data ecosystem.