Data Engineering Podcast

Data sharing across business and platform boundaries is complex due to business rules, regulations, and technical considerations. Andrew Jefferson discusses building a robust system for data sharing, the techno-social considerations, and the Bobsled platform that aims to simplify the process. Topics include challenges of data sharing across cloud platforms, boundaries in data transfer systems, innovative applications of data sharing, shift left and shift right mentality, and the lack of AI and vector database solutions.

The podcast discusses the RisingWave database engine for stream processing on S3, its architecture, challenges faced, and potential integration with a data lakehouse. It explores the use of Kafka for buffering and converting data formats, enhancing Postgres with real-time processing, and the differences in change data capture handling. The episode also covers workflow, onboarding, integration, and unexpected use cases of RisingWave in the manufacturing industry.

Learn about Scanner, a fast querying platform for security log data. Discover the challenges of managing data lakes and the benefits of using a search index. Explore the design philosophies of the Scanner platform and its integration into security log analysis workflows. Understand the indexing strategies for variegated data and the importance of regulatory compliance and data security. Also, find out about the need for better visibility and queryability in data management.

This podcast explores the evolution of database technology, the role of Customer Data Platforms (CDPs) in the business data ecosystem, and architectural approaches to data management. The guest also discusses future plans for ActionIQ and the future of CDPs and data management technology.

The guest, Jignesh Patel, discusses his research on technical scalability and user experience improvements in data management. They explore the challenges of meeting data demand, the limitations of Moore's Law, efficient data retrieval and indexing, strategies for real-world context, and future problems and challenges in complex systems and data processing. The guest also highlights the importance of data discovery in data management technology.

CTO of fintech startup Monite discusses designing and implementing data platforms for the complexities of working with financial data. Topics include data challenges, regulatory requirements, managing backups, machine learning in fintech, reshaping accounting and customer support, and data governance challenges.

Elad Eldor, author of 'Kafka: Troubleshooting in Production', discusses the challenges of operating Kafka at scale and ways to mitigate potential issues. Topics include the importance of Kafka in the data pipeline, doubling retention in Kafka, managed vs. self-managed Kafka clusters, data lake complexity, monitoring for Kafka, troubleshooting unreplicated partitions, the cost of running Kafka in the cloud, and the need for a correlation tool.

The podcast discusses the challenges of the modern data stack and how 5X is pre-integrating the best tools from each layer to solve these issues. It explores the need for a centralized control plane, strategic investments in data capabilities, and the benefits of consolidating to a single solution. The speaker also shares insights on platform building, simplifying user experience, and lessons learned in introducing a new product category.

The podcast discusses the Anomstack project and its goal of providing simple anomaly detection. They explore the definition and prioritization of metrics, implementation and optimization of AMSTAC, and extending the project's capabilities. They also touch on data lakes, Starburst analytics, and challenges in data management.

In this podcast, Andrei Tserakhau discusses the challenges and solutions in data transfer and connectors, handling incremental data transfer and deduplication, building a user-friendly data transfer service, building scalable data engineering systems, building a data transfer product with change data capture and air-bike connectors, and challenges and gaps in data management tooling and technology.