20VC: AI Scaling Myths: More Compute is not the Answer | The Core Bottlenecks in AI Today: Data, Algorithms and Compute | The Future of Models: Open vs Closed, Small vs Large with Arvind Narayanan, Professor of Computer Science @ Princeton
Aug 28, 2024
auto_awesome
Arvind Narayanan, a Princeton Professor and expert in AI and cryptocurrency, debunks the myth that simply adding more compute enhances AI performance. He emphasizes that data quality and algorithms are critical bottlenecks, questioning the future of synthetic data. Arvind draws parallels between AI and crypto hype, expressing skepticism about commoditization. He explores the potential of various small models versus few large ones, while highlighting the ethical implications and challenges in assessing AI's real-world applications and advancements.
Arvind Narayanan argues that adding more compute alone does not guarantee improved AI model performance, as diminishing returns become evident.
The importance of understanding product-market fit in generative AI is highlighted, stressing the need for genuine product development over mere novelty.
Quality of data remains a significant concern in AI development, with a critical focus on effective data gathering over sheer quantity.
Deep dives
The Diminishing Returns of AI Training
The future of large AI models may be constrained by diminishing returns regarding compute performance. Historically, significant increases in model size have led to improved performance, but this trend appears to be plateauing as data becomes a bottleneck. Many companies have already trained models on nearly all accessible data, and while further compute still aids performance, the gains may not be as impactful as before. The current landscape suggests a shift towards smaller models that maintain capability while being cheaper and more manageable.
The Misalignment of AI Product Development
Generative AI companies have faced challenges as they naively anticipated users would discover thousands of applications for AI without significant product development. This misunderstanding has affected their strategies and led to a lack of focus on creating products that truly meet market needs. Developers mistakenly believed the novelty of AI would suffice, ignoring the fundamental principles of product-market fit. This misalignment has highlighted the importance of building tailored solutions rather than leaving it to user discovery.
Emerging Challenges in Data Quality
The conversation around synthetic data and unmined data sources reveals a deeper issue regarding the quality versus quantity of training data. While synthetic data might augment existing datasets, it often fails to introduce genuinely new learning capabilities, leading to a compromise in quality. Furthermore, the vast amount of unmined data, such as video content, does not necessarily translate into useful tokens for training models, making it crucial to prioritize the effectiveness of data gathering. Hence, data quality remains a paramount concern as AI development progresses.
AI Adoption and Organizational Learning
To effectively harness AI in organizations, there needs to be a shift from passive observation to active deployment and iterative learning. This approach allows AI systems to learn from real-world interactions rather than relying solely on pre-existing data. Historically, many learning processes have gone undocumented, making it challenging for models to benefit from organizational knowledge. Organizations must create environments where AI can engage and adapt to actual workflows, essential for bridging the gap between capability and implementation.
The Need for Realistic AGI Expectations
Predictions surrounding artificial general intelligence (AGI) often overstated possibilities based on current AI developments, with many experts maintaining that substantial breakthroughs remain elusive. AGI is frequently framed in terms of automating economically valuable tasks, yet the complexities involved continue to increase as expectations rise. Leaders in AI often project ambitious timelines for AGI, but historical trends reveal that progress is rarely linear and can unexpectedly hit roadblocks. A more practical approach would involve managing expectations and focusing on immediate applications rather than speculative future capabilities.
Arvind Narayanan is a professor of Computer Science at Princeton and the director of the Center for Information Technology Policy. He is a co-author of the book AI Snake Oil and a big proponent of the AI scaling myths around the importance of just adding more compute. He is also the lead author of a textbook on the computer science of cryptocurrencies which has been used in over 150 courses around the world, and an accompanying Coursera course that has had over 700,000 learners.
In Today's Episode with Arvind Narayanan We Discuss:
1. Compute, Data, Algorithms: What is the Bottleneck:
Why does Arvind disagree with the commonly held notion that more compute will result in an equal and continuous level of model performance improvement?
Will we continue to see players move into the compute layer in the need to internalise the margin? What does that mean for Nvidia?
Why does Arvind not believe that data is the bottleneck? How does Arvind analyse the future of synthetic data? Where is it useful? Where is it not?
2. The Future of Models:
Does Arvind agree that this is the fastest commoditization of a technology he has seen?
How does Arvind analyse the future of the model landscape? Will we see a world of few very large models or a world of many unbundled and verticalised models?
Where does Arvind believe the most value will accrue in the model layer?
Is it possible for smaller companies or university research institutions to even play in the model space given the intense cash needed to fund model development?
3. Education, Healthcare and Misinformation: When AI Goes Wrong:
What are the single biggest dangers that AI poses to society today?
To what extent does Arvind believe misinformation through generative AI is going to be a massive problem in democracies and misinformation?
How does Arvind analyse AI impacting the future of education? What does he believe everyone gets wrong about AI and education?
Does Arvind agree that AI will be able to put a doctor in everyone's pocket? Where does he believe this theory is weak and falls down?
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode