The Data Engineering Show Data engineering from the early 2000s till today - BlackRock
Jun 8, 2023
In this conversation, Krishnan Viswanathan, Sr. Director of Engineering at BlackRock with over 20 years in the data space, shares insights from his journey. He reflects on data engineering's evolution, especially the cyclical challenges faced by organizations since the 2000s. Krishnan discusses the transformation from basic reporting tools to sophisticated systems and emphasizes the need for effective data communication across teams. He also highlights the significance of aligning data strategies with business goals for better decision-making.
AI Snips
Chapters
Transcript
Episode notes
Early Cisco Data Journey
- At Cisco in the 1990s, executives got weekly Excel reports and had to phone multiple departments for details.
- Krishnan led a pioneering web-based executive information system that drastically sped up business book closing.
Living Through Dot-com Crash
- Krishnan lived through the dot-com bubble burst firsthand as he handled Cisco's financial forecasting.
- This experience taught him that their reporting engine lacked proactive alerting, inspiring development of a more dynamic model.
The 2010 Data Paradigm Shift
- After 2010, MPP databases broke the old data warehousing paradigm allowing scalable data loading and analytics.
- However, data lakes created chaos without data models, quality controls, and governance to keep data reliable.