Data Engineering Podcast

In this podcast, Andrei Tserakhau discusses the challenges and solutions in data transfer and connectors, handling incremental data transfer and deduplication, building a user-friendly data transfer service, building scalable data engineering systems, building a data transfer product with change data capture and air-bike connectors, and challenges and gaps in data management tooling and technology.

In this podcast, the host discusses the challenges of integrating components in data platform architectures, including user experience, data sharing and delivery, and shadow IT. They explore event-driven pipelines, access control, data flow ownership, and metadata propagation. The importance of reliable integrations and extensible systems is emphasized, along with tools like Open Lineage and DBT. Python and open metadata platforms are highlighted for simplifying integration and managing permissions and roles across data tools.

Learn practical advice for building and scaling dbt projects, including adopting and using dbt, complexities of data lakes, challenges with YAML in dbt projects, scaling dbt projects, and importance of planning and structure in dbt.

Eran Yahav, founder of Tabnine, discusses the journey of building an AI assistant for software engineers. Topics include advancements in AI code completion, the usage and effectiveness of Tabnine, challenges of customizing generative AI for software engineering, and future directions for Tabnine.

Lukas Fittl, a database performance expert, discusses performance bottlenecks in PostgreSQL, tools like 'explain' in PostgreSQL, common optimization challenges, and the importance of tuning configuration parameters. He also shares insights on the development of PG analyze, enabling performance settings in PostgreSQL, and the evolution of database engines.

Tanya Bragin, an experienced product manager for major vendors, shares insights on how to approach tool selection in the database market. Topics include open-source technologies, trends in database market, challenges in data projects, importance of data observability tools, and future trends.

Ranjith Raghunath shares his thoughts on building a strategy for data products, including centralizing vs decentralizing data product strategy, managing technical debt, and the importance of metrics in data product strategy.

This podcast episode discusses the challenges of building and maintaining stream processing infrastructure. It explores the evolution of streaming systems and the role of platforms like Decodable in simplifying stream processing. The speakers also delve into the challenges of stream processing applications, different ways to interact with Decodable, the importance of 'glue' in stream processing, and the biggest gap in data management tooling and technology.

Max Cho, Founder of a business to make policy selection more navigable, discusses the challenges of navigating the opaque insurance industry. Topics include data collection and analysis, automating a manual industry, insurance pricing transparency, challenges of AI navigation, data preprocessing for analysis, understanding policy complexities, and the utility of large language models in the insurance industry.

AI's impact on ETL processes, using generative AI for unstructured data, AI's role in ETL pipelines, experimenting with AI models, evolving role of AI assistants in data engineering, considerations and challenges of using AI in ETL pipelines, changing landscape of ETL tools