Prukalpa Sankar, Co-Founder of Atlan, discusses building trust through data, revolutionizing data discovery with data catalogs, understanding data lineage complexity, balancing value and complexity in data analysis projects, reimagining data governance as data enablement, and advancing data governance in compliance-driven industries.
Read more
AI Summary
AI Chapters
Episode notes
auto_awesome
Podcast summary created with Snipd AI
Quick takeaways
Creating comprehensive data asset profiles improves data understanding and trust.
Data catalogs enhance data discoverability and transparency within organizations.
Strategic metadata management and automation are crucial for effective data governance.
Deep dives
The Origin Story of Atlan and Data Asset Profiles
The journey of Atlan was sparked by the founders' need to solve internal data management challenges within their team. Drawing inspiration from how other teams handle their assets, they envisioned creating simple, comprehensive data asset profiles similar to profiles on GitHub for code and Salesforce for leads. These profiles would provide details on the data's origin, meaning, ownership, lineage, and changes over time.
Challenges in Data Analysis Tooling
While analysis tools have advanced, challenges persist in locating and understanding data, especially within poorly documented databases. Data practitioners often spend significant time finding and interpreting data, impeding their ability to focus on core analysis tasks. The fragility of data workflows further complicates issues, highlighting the need for improved data governance.
The Genesis of Atlan: Solving Internal Data Team Fragility
The founding team's experience with critical data team disruptions led to the inception of Atlan. Emphasizing the need to enhance team resilience, Atlan aimed to streamline data management challenges internally. By drawing parallels from successful tools in other domains like GitHub, the team envisioned creating a robust data asset profile system to address data management needs.
Understanding the Significance of Data Catalogs
Data catalogs serve as a pivotal component in modern data ecosystems, aligning data with organizational asset treatment. Comparable to product pages on e-commerce platforms, a data catalog houses key information on various data assets, enabling users to explore and understand data meaning, lineage, and ownership. By facilitating data discoverability and transparency, data catalogs play a vital role in enhancing data usage and trust.
Empowering Data Teams through Enhanced Metadata Management
Effective data governance relies on strategic metadata management to ensure data assets align with organizational policies and regulations. By integrating metadata control planes, organizations can enforce policies such as data privacy requirements seamlessly throughout the data stack. Utilizing automation and AI tools can further enhance data governance efforts, promoting compliance, auditability, and data asset security.
In the fast-paced work environments we are used to, the ability to quickly find and understand data is essential. Data professionals can often spend more time searching for data than analyzing it, which can hinder business progress. Innovations like data catalogs and automated lineage systems are transforming data management, making it easier to ensure data quality, trust, and compliance. By creating a strong metadata foundation and integrating these tools into existing workflows, organizations can enhance decision-making and operational efficiency. But how did this all come to be, who is driving better access and collaboration through data?
Prukalpa Sankar is the Co-founder of Atlan. Atlan is a modern data collaboration workspace (like GitHub for engineering or Figma for design). By acting as a virtual hub for data assets ranging from tables and dashboards to models & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Slack, BI tools, data science tools and more. A pioneer in the space, Atlan was recognized by Gartner as a Cool Vendor in DataOps, as one of the top 3 companies globally. Prukalpa previously co-founded SocialCops, world leading data for good company (New York Times Global Visionary, World Economic Forum Tech Pioneer). SocialCops is behind landmark data projects including India’s National Data Platform and SDGs global monitoring in collaboration with the United Nations. She was awarded Economic Times Emerging Entrepreneur for the Year, Forbes 30u30, Fortune 40u40, Top 10 CNBC Young Business Women 2016, and a TED Speaker.
In the episode, Richie and Prukalpa explore challenges within data discoverability, the inception of Atlan, the importance of a data catalog, personalization in data catalogs, data lineage, building data lineage, implementing data governance, human collaboration in data governance, skills for effective data governance, product design for diverse audiences, regulatory compliance, the future of data management and much more.