DataTalks.Club
Data Engineering for Fraud Prevention - Angela Ramirez
Oct 6, 2023
Angela Ramirez, a data engineer with experience in fraud prevention, talks about her career journey, the usefulness of knowing ML as a data engineer, best practices for system design and data engineering, working with different types of databases including document and network-based databases, and selecting the appropriate database type to work with. She also discusses the importance of software engineering knowledge in data engineering, data quality check tooling, debugging failed jobs, and working with external data sources.
54:14
Episode guests
AI Summary
Highlights
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Working with external data sources requires establishing data contracts, understanding data reliability, and considering batch versus real-time data.
- Dealing with failed jobs and debugging requires identifying root causes, relying on experience, documentation, and runbooks.
Deep dives
Challenges of Working with External Data Sources
Working with external data sources can present challenges in terms of identifying the right teams to work with, obtaining proper documentation, and ensuring the data remains consistent and accessible. Data engineers must establish data contracts and understand the frequency and reliability of the data received. Additionally, considerations such as batch versus real-time data and potential changes to the data source must be taken into account.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
