Letter from Reykjavik: Genomics, Chess, Hyperscaling genAI, and Quantum Black Holes — #67
Aug 29, 2024
auto_awesome
Bobby Fischer, the legendary chess player, leaves a lasting legacy as the podcast explores his influence on the game and its connection to scientific pursuits. Reflect on the innovative work at deCODE Genetics in Reykjavik and how it reshapes big data genomics. Delve into the intersection of chess and AI, discussing the challenges of scaling models and the limits of current architectures. The conversation also touches on the quest for artificial general intelligence, highlighting the intriguing interplay between science and intellectual disciplines.
Decode Genetics' innovative approach to genomics highlights the potential of genealogical data to infer the genetic makeup of an entire population.
Concerns over the effectiveness of synthetic data for training AI models raise questions about the depth needed for complex understanding in various fields.
Deep dives
Challenges of Synthetic Data in AI
There are concerns regarding the effectiveness of synthetic data for training large language models, especially in complex areas like finance, law, and history. The risk is that synthetic data may not reflect the depth and nuances captured in the first few trillion tokens generated by human writers. Thus, as the reliance on synthetic data increases for future AI training, its potential shortcomings could hinder the models' understanding of intricate concepts tied to these fields. Without robust human-generated data, the risk grows that the synthetic models may lack the depth and accuracy needed for meaningful analysis.
Experiences in Reykjavik and Decode Genetics
During a visit to Reykjavik, significant insights were gained from engaging with the local scientific community, particularly at Decode Genetics, known for its unique approach to genomics. By genotyping a substantial portion of the Icelandic population, Decode was able to infer the genetic makeup of the entire population based on genealogical data. This innovative method has allowed Decode to produce numerous impactful publications, underscoring the value of their dataset despite some financial setbacks in the past. Additionally, the historical context of the company highlights the challenges and ambitions in genetic research in pursuit of understanding complex diseases.
Chess and Its Uniqueness in Intellectual Pursuits
The podcast reflects on the evolution of chess as both a competitive sport and an intellectual pursuit, triggered by the 1972 World Chess Championship in Reykjavik. The speaker’s personal narrative highlights a critical evaluation of chess compared to other disciplines, such as physics and mathematics, suggesting that the intellectual return on investment in chess is limited compared to broader academic fields. While chess has garnered renewed interest among youth today, its classification as an 'artificial game' raises questions about its long-term value in education and intellectual exploration. This perspective is further enriched by anecdotes from the speaker's interactions with peers who have navigated both chess and academia.
Future of AI and Hyper Scaling
The discussion around the hyper-scaling of AI underscores the considerable financial investments being made to improve large language models, with spending nearing a significant percentage of US GDP. The current trajectory involves increasing the scale of models with the expectation of emerging capabilities, but the reliance on synthetic data raises concerns about maintaining optimal progress. A complex relationship exists among the size of the model, data requirements, and computational resources needed for effective training. The ongoing exploration into different model architectures and techniques suggests that while near-term advancements may be achieved, truly transforming AI to human-like adaptability may require substantial foundational breakthroughs.
24:26 Is the Transformer architecture enough for AGI?
29:45 Quantum black holes
Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.
--
Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (SuperFocus.ai, SafeWeb, Genomic Prediction, Othram) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU. Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on X @hsu_steve.
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode