What is the state of Data engineering today and where is it going (or should it be going)? Who better to talk about Data Engineering than the authors of the recent O’Reilly book “Fundamentals of Data Engineering”, Joe Reis and Matt Housley from Ternary Data.
Join Tim, Juan, Joe and Matt to discuss the state of data engineering.
Conversation highlights:
- [00:44] Introduction to Joe Reis and Matt Housley
- [03:24] Warm up: what have you engineered or built that has gone terribly wrong?
- [08:28] Data engineering tools and magpie syndrome
- [13:15] How often and where do you see data teams stumbling to get the value out of the technology they have?
- [17:08] Thoughts on diving deep into technologies where there is a skills gap
- [19:39] The curse of familiarity in the context of tools
- [23:55] Learning outside the context of an enterprise, and continously educating yourself on data
- [27:23] Data modeling and where to start learning
- [30:48] Data modeling = business concepts and definitions to relations among data
- [32:05] The human aspect and technical side of data modeling
- [36:19] Business literacy for data engineers
- [38:39] The lifecycle of data
- [43:02] Analytics engineering, the knowledge scientists
- [46:09] Roles within data engineering and bifurcation
- [48:26 The future of data engineering and tabular data
- [52:12] Auto Machine Learning and Tabular Data
- [55:00] Lightning round
- [01:01:01] Tim and Juan's takeaways
- [01:06:18] Three questions about data, life, resources and the show's next guest
