Handling fuzzy relationships and causality limits

Summary
In this episode of the Data Engineering Podcast Vijay Subramanian, founder and CEO of Trace, talks about metric trees - a new approach to data modeling that directly captures a company's business model. Vijay shares insights from his decade-long experience building data practices at Rent the Runway and explains how the modern data stack has led to a proliferation of dashboards without a coherent way for business consumers to reason about cause, effect, and action. He explores how metric trees differ from and interoperate with other data modeling approaches, serve as a backend for analytical workflows, and provide concrete examples like modeling Uber's revenue drivers and customer journeys. Vijay also discusses the potential of AI agents operating on metric trees to execute workflows, organizational patterns for defining inputs and outputs with business teams, and a vision for analytics that becomes invisible infrastructure embedded in everyday decisions.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
• Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
• Your host is Tobias Macey and today I'm interviewing Vijay Subramanian about metric trees and how they empower more effective and adaptive analytics

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you describe what metric trees are and their purpose?
• How do metric trees relate to metric/semantic layers?
• What are the shortcomings of existing data modeling frameworks that prevent effective use of those assets?
  • How do metric trees build on top of existing investments in dimensional data models?
• What are some strategies for engaging with the business to identify metrics and their relationships?
• What are your recommendations for storage, representation, and retrieval of metric trees?
• How do metric trees fit into the overall lifecycle of organizational data workflows?
• When creating any new data asset it introduces overhead of maintenance, monitoring, and evolution. How do metric trees fit into existing testing and validation frameworks that teams rely on for dimensional modeling?
  • What are some of the key differences in useful evaluation/testing that teams need to develop for metric trees?
• How do metric trees assist in context engineering for AI-powered self-serve access to organizational data?
• What are the most interesting, innovative, or unexpected ways that you have seen metric trees used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on metric trees and operationalizing them at Trace?
• When is a metric tree the wrong abstraction?
• What do you have planned for the future of Trace and applications of metric trees?

Contact Info

• LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

• Metric Tree
• Trace
• Modern Data Stack
• Hadoop
• Vertica
• Luigi
• dbt
• Ralph Kimball
• Bill Inmon
• Metric Layer
• Dimensional Data Warehouse
• Master Data Management
• Data Governance
• Financial P&L (Profit and Loss)
• EBITDA ==Earnings before interest, taxes, depreciation and amortization

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA