#146 - ChatGPT’s 1 year anniversary, DeepMind GNoME, Extraction of Training Data from LLMs, AnyDream
Dec 12, 2023
auto_awesome
DeepMind guests celebrate ChatGPT's 1 year anniversary. They discuss DeepMind GNoME and extraction of training data from LLMs. Also, they cover topics such as AI-powered tax prep, AI image generator, and AI-powered gift finder. OpenAI news includes board composition changes, purchase of AI chips, and rival high valuation. Google's Gemini AI launch is delayed. Other topics include AI acquisitions, open source language models, and scaling molecular modeling with deep learning. They also mention the Pentagon's Replicator program and concerns over nonconsensual deep fakes.
DeepMind's Gnome project uses deep learning to predict new crystal structures, significantly expanding the catalog of known stable crystals.
Open-source language models like Llama2 and DeepSeek are showing promising results and narrowing the gap with closed-source models.
A study by DeepMind quantified the carbon footprint of AI tasks, revealing higher environmental impact in image generation compared to text generation.
Deep dives
DeepMind's Gnome project expands catalog of known stable crystals
DeepMind has developed the Gnome project, a custom model that uses deep learning to predict potential new crystal structures. By generating different atomic structures and using a graph neural network to model their properties, Gnome has significantly expanded the catalog of known stable crystals. The model has shown promising results, outperforming previous methods and increasing the number of computationally stable materials by an order of magnitude.
Open-source LLMs closing the gap with closed-source models
Open-source language models are making progress in catching up to closed-source models like ChatGPT. While there are still challenges, such as changing performance, unknown training data, and higher costs, open-source models like Llama2 and DeepSeek are showing promising results in various benchmarks, including language comprehension, coding, and math. The trend of open-source models improving and scaling suggests a narrowing divergence between closed-source and open-source models.
Google's DeepMind quantifies carbon footprint of AI image generation
A recent study by DeepMind has quantified the carbon footprint of various AI tasks, including image generation, object detection, text classification, and more. The study found that tasks like image generation and image captioning have a higher carbon footprint, requiring significant computational resources. On the other hand, text generation has a lower carbon footprint compared to image-based tasks. This research provides important insights into the environmental impact of AI and can guide efforts to develop more energy-efficient models.
Research highlights potential of deep learning for materials discovery
Researchers at DeepMind have developed a deep learning model called Gnome that aims to expand the catalog of known stable crystals. By using a combination of data sets and a graph neural network, Gnome can predict potential new crystal structures. The model has shown promising results, outperforming previous methods and significantly increasing the number of computationally stable materials. This research has significant implications for materials discovery and showcases the potential of deep learning in advancing scientific research.
Adopting Ethical Principles for Generative AI in Healthcare
A collaboration of different backgrounds has put forward a set of ethical principles for generative AI in healthcare, along with a framework for adopting and expanding these principles practically.
Low-Dimensional Gaussian Mixtures and Transformers
Researchers have developed a mathematical definition for low-dimensional Gaussian mixtures, allowing them to recreate transformers based on this ideal. This understanding can enhance algorithmic efficiency and deepen our comprehension of multi-headed self-attention.