Dr. Emre Kiciman from Microsoft Research discusses causal machine learning vs correlational, his DoWhy library, key inference steps, future of AGI, leveraging social media for social issues, recommended software tools, hiring criteria for researchers.
Causal ML emphasizes causal assumptions to infer cause and effect relationships in decision-making scenarios.
Emre's DoWhy library aids in modeling assumptions, estimating causal effects, and improving confidence in causal inference outcomes.
Causal inference involves key steps like modeling assumptions, identification, estimation, and validation/refutation to ensure robust analysis outcomes.
Deep dives
Causal Machine Learning and its Impacts
Causal machine learning focuses on incorporating domain knowledge to infer cause and effect relationships, emphasizing the importance of causal assumptions. It aims to guide machine learning models to pay attention to the right patterns through stability in cause and effect relationships, differentiating it from conventional machine learning that looks for correlations only. The application of causal ML in decision-making scenarios is highlighted for robustness against data distribution shifts caused by changing policies.
PiY Open Source Library for Causal Machine Learning
The PyY open source library initiated by Dr. Emre Keczimun and Amit Sharma aims to facilitate the understanding and practical application of causal inference methods. It fills the gap in existing tools by providing an end-to-end scaffolding for modeling assumptions, estimating causal effects, and validating and refuting assumptions. The library focuses on helping users reason over assumptions and improve confidence in causal inference outcomes.
The Four Key Steps of Causal Inference
Causal inference involves four key steps: modeling assumptions, identification, statistical estimation, and validation/refutation. Modeling assumptions include capturing domain knowledge to guide analysis, while identification focuses on finding strategies to calculate causal effects. Statistical estimation employs various methods suitable for different data scales and types. Validation/refutation involves testing assumptions made throughout the process to enhance confidence in the analysis outcomes.
Applications of Causal Inference in Agriculture and Health
The application of causal machine learning in agriculture, particularly in soil health management for carbon sequestration, is highlighted as a promising area with potential global impact. Additionally, the use of causal methods in healthcare, especially to enhance the traditional randomized control trial process for treatment development, shows significant potential for advancements in healthcare research. Partnerships in different sectors like agriculture and healthcare aim to leverage causal methods for impactful outcomes.
Influence of Domain Knowledge in Causal Machine Learning
Causal machine learning heavily relies on domain knowledge and causal assumptions made during the analysis process to ensure robust and accurate results. The integration of causal methods with deep learning models by imposing constraints consistent with causal graphs aims to enhance pattern recognition and generalizability across various data types, including unstructured text data. Research efforts continue to focus on advancing causal inference techniques for broader and more effective applications in the future.
Dr. Emre Kiciman, Senior Principal Researcher at Microsoft Research joins the podcast to share his world-leading knowledge on causal machine learning.
This episode is brought to you by Datalore (datalore.online/SDS), the collaborative data science platform, and by Zencastr (zen.ai/sds), the easiest way to make high-quality podcasts. Interested in sponsoring a SuperDataScience Podcast episode? Visit JonKrohn.com/podcast for sponsorship information.
In this episode you will learn: • What is causal machine learning? [5:52] • Causal machine learning vs correlational machine learning [10:10] • Emre’s DoWhy open-source library [16:17] • The four key steps of causal inference [21:24] • How and why Emre’s key steps of causal inference will impact ML [26:36] • Emre's thoughts on the future of causal inference and AGI [34:09] • How Emre leverages social media data to solve social problems [38:36] • What's next for Emre's research [46:02] • The software tools Emre highly recommends [55:16] • What he looks for in the data science researchers he hires [58:45]