Episode 110: Wisdom in Building Data Infrastructure, Lessons From Open-Source Development, The Missing README, and The Future of Data Engineering with Chris Riccomini
Datacast
00:00
Evolution of Data Infrastructure at WePay
The chapter explores the journey of transitioning from a monolith to a service-based architecture at WePay and the challenges faced in evolving the data warehouse to meet growing needs, discussing the adoption of Airflow and BigQuery. It underscores the importance of early investment in data quality and delves into various methods like traditional checks and machine learning, with a nod to Anomolo for specialized data quality using ML.
Play episode from 01:00:43
Transcript
