Understanding Image Compression through Semantic Learning

Can AI be applied to enhance geospatial data for climate, nature and people? This episode kicks off a miniseries about domain-specific foundation models. Following the trends in language processing, domain-specific foundation models are enabling new possibilities for a variety of applications, including Earth observation. During this conversation, I am joined by Bruno Sánchez-Andrade Nuño, Executive Director of Clay, a nonprofit organization harnessing the power of AI for satellite images, spatial data, and more. Bruno shares the functionality and concept behind Clay, and his journey to building it. He goes on to unpack the tool’s foundation model in broad strokes, before explaining why it's important, and sharing the challenges he has faced along the way. We discuss the legal aspects of building Clay, and it’s primary goal to make it as easy as possible for any user to achieve their goals. We also touch on what the future might hold for Clay and the future of Earth observation. Thanks for listening!

Key Points:

• Introducing guest, Bruno Sánchez-Andrade Nuño, Executive Director at Clay.
• His journey from NASA astrophysicist to climate change, social development, and AI researcher.
• What Clay focuses on: using remote sensing maps to interpret the Earth’s data.
• The mechanics of how Clay is used and how different feature sets compare to one another.
• A broad explanation of the tool’s foundation model and why it is quicker, cheaper, and more environmentally friendly.
• Two main benefits of the tool that Bruno finds most exciting. 
• Data and infrastructure required to build Clay including 70 million satellite and aerial images.
• Measuring what the model understands and the process of compressing an image into 700 numbers.
• Privacy and intellectual property in the realm of satellite imaging and mapping. 
• What commercial imagery could add to the model and how it might be integrated in the future. 
• Clay’s partnerships with university and company groups
• Why the focus of Clay is to make it as easy as possible for anyone to use the tool for anything they want to do. 
• Challenges encountered on the road to building Clay: explaining what it is.
• The complexity of benchmarking foundation models and how this relates to Clay. 
• Working with partners to build Clay and the rest of the ecosystem. 
• Lessons from building Clay that may apply to other foundation models.
• Bruno’s predictions for the future of foundation models and Clay. 
• What is certain about the future of Clay and our understanding of Earth. 

Quotes:

“Clay is trying to figure out how to finally increase the adoption of remote sensing by leveraging a tool that itself is very complex, but the result of that tool is very easy to use.” — Bruno Sánchez-Andrade Nuño

“If you start with a foundational model that gets you most of the way there, [then] you can create those trials much quicker, much cheaper, and much more environmentally friendly.” — Bruno Sánchez-Andrade Nuño

“This is so new, we get the chance, those of us working on it, that we can save the whole industry, if you will, the whole space of AI for it.” — Bruno Sánchez-Andrade Nuño

“Clay, I believe, is not only the largest and most efficient model AI for Earth, for any kind of like foundational model. It is also completely open source.” — Bruno Sánchez-Andrade Nuño

“What we try to focus on is how can we make it as simple as possible for anyone anywhere to use this model for anything they want to do.” — Bruno Sánchez-Andrade Nuño

Links:

Bruno Sánchez-Andrade

Bruno Sánchez-Andrade Nuño on X

Bruno Sánchez-Andrade Nuño on LinkedIn

Clay

Clay on LinkedIn

Resources for Computer Vision Teams:

LinkedIn – Connect with Heather.

Computer Vision Insights Newsletter – A biweekly newsletter to help bring the latest machine learning and computer vision research to applications in people and planetary health.

Computer Vision Strategy Session – Not sure how to advance your computer vision project? Get unstuck with a clear set of next steps. Schedule a 1 hour strategy session now to advance your project.