[30] Dustin Tran - Probabilistic Programming for Deep Learning
Aug 14, 2021
auto_awesome
Dustin Tran, a research scientist at Google Brain, specializes in probabilistic programming and deep learning. He discusses his PhD thesis on integrating probabilistic modeling with deep learning, highlighting the innovative Edward library and new inference algorithms. The conversation dives into the evolution of AI tools like TensorFlow, emphasizing their democratizing impact. Dustin also shares insights on transitioning from PhD to research roles, the importance of addressing uncertainty in neural networks, and the balance between academic benchmarks and practical advancements.
Deep probabilistic programming, exemplified by Dustin Tran's Edward library, blends probabilistic models with deep learning for complex data analysis.
The design of Edward emphasizes inference as a central focus, fostering a more integrated relationship between models and their analysis methods.
Open-source contributions, while demanding, yield significant rewards by enhancing collaboration and providing vital tools for the research community.
Deep dives
Deep Probabilistic Programming
Deep probabilistic programming combines probabilistic modeling with deep learning to create robust systems for complex data analysis. It involves the development of software that supports the implementation of probabilistic models and inference algorithms, such as Edward, a Python library designed by Dustin Tran during his PhD. Edward is notable for enabling users to scale up models and leverage deep neural networks while maintaining rigorous probabilistic principles. Its evolution into Edward 2.0 further emphasizes the integration of flexible inference methods and automatic differentiation, allowing for more sophisticated probabilistic programming applications.
Engineering versus Research Mindsets
The intersection of engineering, research, and science presents unique challenges and opportunities within the realm of machine learning. Dustin Tran emphasizes the important relationship between well-engineered software and the advancement of research, suggesting that effective library design can significantly impact the research outcomes. The availability of tools such as TensorFlow and PyTorch has lowered the barriers for researchers to experiment with new ideas, showcasing the crucial role of engineering in accelerating innovative research. Understanding this dynamic allows researchers to develop meaningful systems that address real-world applications and advance scientific knowledge.
The Design of Edward and Edward 2.0
The design principles guiding the development of Edward and its successor, Edward 2.0, hinged on treating inference as a primary consideration rather than a secondary task. This approach fosters an environment where models are developed hand-in-hand with the inference methods needed to analyze them, enabling researchers to flexibly adapt their models. The iterative nature of this design process allows for continuous improvement and innovation, ensuring that new inference methods can be seamlessly integrated into the library. By focusing on the underlying systems architecture, Edward facilitates a wide array of applications across various domains.
Variational Inference and Gaussian Processes
Variational inference methods, particularly the variational Gaussian process model developed by Dustin Tran, highlight the integration of deep learning with traditional statistical methodologies. This model aims to provide flexible posterior approximations, enhancing the effectiveness of inference in complex applications. By challenging the mean field assumptions commonly used in earlier models, this work demonstrates how hierarchical Bayesian structures can be employed to capture more nuanced relationships within datasets. The introduction of auxiliary variables represents a significant advancement in allowing for richer representations of uncertainty in probabilistic models.
Navigating Open Source and Impactful Research
Dustin Tran's experiences with maintaining and developing open-source libraries like Edward reveal insights into the challenges and rewards of contributing to the research community. Open source development requires ongoing commitment to maintain code, respond to user queries, and incorporate feedback, often without direct academic recognition. However, the impact of providing useful tools for other researchers can be immensely rewarding, showcasing the value of collaboration and creativity in the scientific process. As Tran acknowledges, finding significant problems to solve and facilitating meaningful research collaborations is essential for long-term success in the field.
Dustin Tran is a research scientist at Google Brain. His research focuses on advancing science and intelligence, including areas involving probability, programs, and neural networks.
Dustin’s PhD thesis is titled "Probabilistic Programming for Deep Learning", which he completed in 2020 at Columbia University.
We discuss the intersection of probabilistic modeling and deep learning, including the Edward library and the novel inference algorithms and models that he developed in the thesis.
- Episode notes: https://cs.nyu.edu/~welleck/episode30.html
- Follow the Thesis Review (@thesisreview) and Sean Welleck (@wellecks) on Twitter
- Find out more info about the show at https://cs.nyu.edu/~welleck/podcast.html
- Support The Thesis Review at www.patreon.com/thesisreview or www.buymeacoffee.com/thesisreview
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
