PhD students Sandeep & Oindrila discuss challenges in creating 3D animals from 2D data, including quality control, training datasets, tie-in with iNaturalist. They explore diffusion models, control nets, and Hollywood FX potential.
Generative AI struggles with creating realistic 3D animals due to limited datasets.
Researchers leverage generative AI and diffusion to predict 3D images from various angles.
Innovation involves control nets for manipulating poses and ensuring anatomical consistency in 3D animal models.
Deep dives
Introduction to You Dream Project and Its Visual Approach
The conversation starts by introducing the You Dream project, focusing on generating anatomically controlled 3D animals using generative AI. The team emphasizes the visual aspect by discussing the need for a link or search term for a rich visual archive paper or related visuals. An announcement for a co-hosting opportunity in graph theory is made, highlighting the project's interdisciplinary nature.
Challenges of 2D to 3D Image Conversion for Animals
The podcast delves into the challenges faced in converting 2D to 3D images for animals due to limited animal photo datasets compared to objects or humans. The diversity and scarcity of animal photos pose a significant obstacle. The team discusses leveraging generative AI and diffusion to predict images from various angles, elucidating the concept of denoising in generative image algorithms.
Innovations in Generating 3D Animal Models
The podcast explores the innovation of generating 3D animal models using control nets to manipulate poses and achieve anatomical consistency. The discussion touches on the process of distilling flesh onto skeletons, incorporating new techniques for fine-tuning and regularization, and ensuring consistent and accurate views of generated 3D assets.
Potential Applications in Industry and VFX Creation
The conversation looks ahead to potential applications of the research in industry, particularly VFX creation for movies. The team discusses the automated creation of 3D animal poses and potential use in major studios for character design. They highlight the versatility and potential impact of the project in art and visual effects.
Future Directions and Career Plans
The podcast delves into the future directions and career aspirations of the researchers post-PhD. Discussions revolve around animating assets, working with iNaturalist datasets, and the need for innovation in human and animal pose animation. The team aims to create tools for VFX creators, explore animation techniques, and enhance asset control for artists.
Generative AI can struggle to create realistic animals and 2D representations often have mistakes like extra limbs and tails. If 2D wasn’t hard enough, there are researchers working on generative 3D models. 3D models present an extra challenge because there is paucity of training datasets.In this episode, PhD students Sandeep and Oindrila walked us through their work on creating 3D animals using 2D data. Join us to learn about their pipelines, quality control, tie in with iNaturalist, and how this tech could streamline FX pipelines.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
