AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Lex Fridman Podcast
#387 – George Hotz: Tiny Corp, Twitter, AI Safety, Self-Driving, GPT, AGI & God
Jun 30, 2023
03:15:19
AI Summary
Highlights
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Tiny Grad aims to challenge the centralized power of companies like NVIDIA and ensure decentralized computation in AI chips.
- The concept of AI relationships brings up ethical and moral questions that would require societal adaptation.
- Decentralized power is crucial in AI accelerators, with a focus on simplifying porting new ML accelerators quickly.
- Diversifying GPUs and reducing centralized power in the industry is important, with AMD being mentioned as a potential alternative.
- The creation of the TinyBox offers a compact, energy-efficient solution for real-time training and inference in machine learning applications.
Deep dives
The Power of Laziness in TinyGrad
Tiny Corp, led by the founder George Hott, developed TinyGrad, a neural network library that follows a risk architecture. TinyGrad removes Turing completeness from each layer of the stack, providing the ability to reason about computations. It defines four types of operations: unary, binary, reduce, and movement ops. Unlike other libraries, TinyGrad does not have a primitive operator for matrix multiplication, instead relying on a combination of other operations. The developer experience with TinyGrad is similar to PyTorch, but offers enhanced visibility into GPU kernels. The goal of Tiny Corp is to challenge the centralized power of companies like NVIDIA to ensure decentralized computation and prevent the nationalization of AI chips.
The Future of AI Relationships
George Hott discusses the concept of AI relationships and the potential loneliness that could drive individuals to seek companionship from AI models. He acknowledges that the boundaries and implications of AI relationships would differ for each person and relationship. Hott expresses his personal wish for AI friends who are more intelligent than he is while acknowledging the ethical and moral questions that arise in this evolving field. He discusses the potential impact of AI relationships on traditional monogamy and suggests that societal views and norms will need to adapt to this new aspect of human experience.
The Role of AI Accelerators and Tiny Grad
George Hott emphasizes the significance of decentralized power in the field of AI accelerators. He highlights the challenges faced by companies like Graphcore, Tensort, and others in building AI chips and software ecosystems. Hott asserts that software is the key focus and that Tiny Grad aims to simplify the process of porting new ML accelerators quickly. He also predicts a potential pivot for Tensort towards risk five CPUs as AI accelerators are essentially a software problem rather than a hardware one. He mentions the importance of writing a performance stack for NVIDIA GPUs before considering the development of specialized AI chip hardware.
Decentralized Power and the Future of GPUs
The podcast episode explores the importance of diversifying GPUs and extending decentralization to other areas. The speaker emphasizes the need to reduce centralized power and highlights the potential dangers of monopolistic control in the GPU industry. AMD is mentioned as a potential alternative to NVIDIA, with a focus on performance and stability.
The TinyBox: A Powerful AI Compute Solution
The podcast discusses the creation of the TinyBox, a high-performance deep learning box capable of running large AI models. The TinyBox is designed to be compact, energy-efficient, and easily deployable in a home setting. It features powerful GPUs, high GPU RAM, and fast drive read bandwidth. The speaker highlights the potential of the TinyBox in enabling real-time training and inference for machine learning applications.
Open Source, AI Safety, and the Future of the Internet
The podcast delves into the role of open source in AI development, with a focus on the risks and benefits associated with AI safety. The speaker advocates for open sourcing model architectures and breakthroughs while acknowledging the need to protect proprietary model weights. They express concern about centralized control and promote the idea of decentralized power as a means to combat potential harm. The discussion also touches on the transformative effects of the internet and the importance of preserving its decentralized spirit.
The Importance of Simplifying Code and Refactoring
In the podcast, the speaker emphasizes the significance of simplifying code and prioritizing refactoring. They discuss how the Twitter codebase could benefit from a refactor to reduce complexity and remove unnecessary code. The speaker suggests implementing comprehensive tests and focusing on making the codebase more modular and easier to understand. They argue that a refactor could lead to faster development and improved productivity, though they acknowledge the challenges of refactoring a large-scale and live service like Twitter.
The Role of Technical Leadership and Building Trust in Codebases
The speaker highlights the importance of technical leadership in maintaining high-quality codebases. They emphasize the need for managers and leaders to have deep technical understanding and the ability to recognize skill in software development. They propose a culture of simplicity and continually improving code, prioritizing refactoring over adding new features. Additionally, the speaker suggests integrating comprehensive testing and implementing tools that maximize the quality of the coding experience for individuals. They also discuss the challenges of integrating tests into a codebase with minimal existing tests but highlight the benefits of building trust in code changes and improving development speed.
The Future of AI in Programming
George discusses the potential benefits of using artificial intelligence and machine learning in programming. He expresses his desire for AI-powered tools that can provide feedback and simplify code, while also highlighting the importance of maintaining control and ensuring the alignment of AI with human values and goals.
The Role of AI in Gaming and Virtual Reality
George shares his passion for computer gaming and virtual reality, emphasizing the potential of intelligent AI to enhance gaming experiences. He envisions a future where AI-powered NPCs in games have advanced conversational abilities and can dynamically interact with players, ultimately creating more immersive and captivating virtual worlds.
George Hotz is a programmer, hacker, and the founder of comma-ai and tiny corp. Please support this podcast by checking out our sponsors:
– Numerai: https://numer.ai/lex
– Babbel: https://babbel.com/lexpod and use code Lexpod to get 55% off
– NetSuite: http://netsuite.com/lex to get free product tour
– InsideTracker: https://insidetracker.com/lex to get 20% off
– AG1: https://drinkag1.com/lex to get 1 year of Vitamin D and 5 free travel packs
Transcript: https://lexfridman.com/george-hotz-3-transcript
EPISODE LINKS:
George’s Twitter: https://twitter.com/realgeorgehotz
George’s Twitch: https://twitch.tv/georgehotz
George’s Instagram: https://instagram.com/georgehotz
Tiny Corp’s Twitter: https://twitter.com/__tinygrad__
Tiny Corp’s Website: https://tinygrad.org/
Comma-ai’s Twitter: https://twitter.com/comma_ai
Comma-ai’s Website: https://comma.ai/
Comma-ai’s YouTube (unofficial): https://youtube.com/georgehotzarchive
Mentioned:
Learning a Driving Simulator (paper): https://bit.ly/42T6lAN
PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
YouTube Full Episodes: https://youtube.com/lexfridman
YouTube Clips: https://youtube.com/lexclips
SUPPORT & CONNECT:
– Check out the sponsors above, it’s the best way to support this podcast
– Support on Patreon: https://www.patreon.com/lexfridman
– Twitter: https://twitter.com/lexfridman
– Instagram: https://www.instagram.com/lexfridman
– LinkedIn: https://www.linkedin.com/in/lexfridman
– Facebook: https://www.facebook.com/lexfridman
– Medium: https://medium.com/@lexfridman
OUTLINE:
Here’s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time.
(00:00) – Introduction
(08:04) – Time is an illusion
(17:44) – Memes
(20:20) – Eliezer Yudkowsky
(32:45) – Virtual reality
(39:04) – AI friends
(46:29) – tiny corp
(59:50) – NVIDIA vs AMD
(1:02:47) – tinybox
(1:14:56) – Self-driving
(1:29:35) – Programming
(1:37:31) – AI safety
(2:02:29) – Working at Twitter
(2:40:12) – Prompt engineering
(2:46:08) – Video games
(3:02:23) – Andrej Karpathy
(3:12:28) – Meaning of life
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode