The Impact of AI on Data Engineering

As the Field CTO & Head of Strategy @ VAST Data, Colleen Tartow, Ph.D., has a vast resume of building data engineering teams from scratch and beyond. Colleen discusses the necessary components for developing new or reorienting existing data programs, strategies for effective communication & collaboration between data & eng functions, the implications of AI technology on data engineering, and integrating cross-functional partners into the data eng planning process & road map. Plus Colleen shares about building the hiring process for data eng functions, when the “data engineering” term or role didn’t exist yet, and how you can apply that to other emerging or undefined functions!

ABOUT COLLEEN TARTOW

Colleen Tartow, Ph.D. is Field CTO and Head of Strategy at VAST Data and has 20+ years of experience in data, analytics, engineering, and consulting. Adept at assisting organizations in deriving value from a data-driven culture, she has successfully led diverse data, engineering, and analytics teams through the development of complex global data management solutions and architecting enterprise data systems. Her demonstrated excellence in data, engineering, analytics, and diversity leadership makes her a trusted senior advisor among executives. An experienced speaker, author, valued mentor and startup advisor, Colleen holds degrees in astrophysics and lives in Massachusetts.

"Everyone wants to be data driven, right? Like no one's going to say, 'No, we don't want data. We just want to function with opinions.' Like nobody's actually going to say that. But that said, getting started on that can be really challenging...

With anything, you have to go back to what does the business really need. Going back to the revenue drivers and the business pain points that you're going to help solve, whether it's monetizing your data directly or using data as an enablement function to actually help in other areas and so I think getting the organization to understand that data is a product of the business and then sort of working back from there into what does that specifically mean.”

- Colleen Tartow   

Interested in joining an ELC Peer Group?

ELCs Peer Groups provide a virtual, curated, and ongoing peer learning opportunity to help you navigate the unknown, uncover solutions and accelerate your learning with a small group of trusted peers.

Apply to join a peer group HERE: sfelc.com/peerGroups

SHOW NOTES:

• Colleen’s experience building a data program from scratch (2:25)
• What it used to be like building a data engineering team (4:43)
• Narrowing to first principles when hiring for / building a data eng team (6:44)
• Frameworks to advocate for more resources to build your org’s data function (7:53)
• Knowing when you need to transition your data side project to a full data program (10:11)
• Building data teams from a zero to one perspective (13:05)
• What “onboarding as discovery” conversations look like (14:38)
• Joining an existing team to implement a defined data-focused function (16:14)
• How to have effective conversations & collaborate with other eng functions (19:19)
• Prioritization strategies when refocusing / creating the data eng org roadmap (21:20)
• How to integrate cross-functional partners into the data eng planning process (22:51)
• The implication of AI on data teams & its intersection with eng teams (24:09)
• Colleen’s decision-making framework (27:54)
• Recommendations for tackling complex data pipelines in different ways (29:27)
• Navigating the paradigm of AI & data eng’s impact on other eng orgs (31:31)
• What the ideal collaboration between data & eng looks like (34:01)
• Recommendations for dealing with points of friction (35:21)
• Steps for aligning data & eng under the same goals (37:16)
• Rapid fire questions (39:04)

LINKS AND RESOURCES

• The Lioness of Boston - Emily Franklin’s deeply evocative novel of the life of Isabella Stewart Gardner, a daring visionary who created an inimitable legacy in American art and transformed the city of Boston itself.

This episode wouldn’t have been possible without the help of our incredible production team:

Patrick Gallagher - Producer & Co-Host

Jerry Li - Co-Host

Noah Olberding - Associate Producer, Audio & Video Editor https://www.linkedin.com/in/noah-olberding/

Dan Overheim - Audio Engineer, Dan’s also an avid 3D printer - https://www.bnd3d.com/

Ellie Coggins Angus - Copywriter, Check out her other work at https://elliecoggins.com/about/