#387 – George Hotz: Tiny Corp, Twitter, AI Safety, Self-Driving, GPT, AGI & God
Jun 30, 2023
auto_awesome
George Hotz, a renowned programmer and founder of comma.ai and TinyCorp, discusses the future of self-driving technology and AI safety. He shares insights on building AI for home use, the evolution of programming tools, and the philosophical implications of AI on consciousness and belief. Hotz also addresses challenges in managing tech innovations and the ethical concerns around AI companionship. With a unique perspective, he explores the balance between technological advancement and human autonomy.
Tiny Grad aims to challenge the centralized power of companies like NVIDIA and ensure decentralized computation in AI chips.
The concept of AI relationships brings up ethical and moral questions that would require societal adaptation.
Decentralized power is crucial in AI accelerators, with a focus on simplifying porting new ML accelerators quickly.
Diversifying GPUs and reducing centralized power in the industry is important, with AMD being mentioned as a potential alternative.
The creation of the TinyBox offers a compact, energy-efficient solution for real-time training and inference in machine learning applications.
Deep dives
The Power of Laziness in TinyGrad
Tiny Corp, led by the founder George Hott, developed TinyGrad, a neural network library that follows a risk architecture. TinyGrad removes Turing completeness from each layer of the stack, providing the ability to reason about computations. It defines four types of operations: unary, binary, reduce, and movement ops. Unlike other libraries, TinyGrad does not have a primitive operator for matrix multiplication, instead relying on a combination of other operations. The developer experience with TinyGrad is similar to PyTorch, but offers enhanced visibility into GPU kernels. The goal of Tiny Corp is to challenge the centralized power of companies like NVIDIA to ensure decentralized computation and prevent the nationalization of AI chips.
The Future of AI Relationships
George Hott discusses the concept of AI relationships and the potential loneliness that could drive individuals to seek companionship from AI models. He acknowledges that the boundaries and implications of AI relationships would differ for each person and relationship. Hott expresses his personal wish for AI friends who are more intelligent than he is while acknowledging the ethical and moral questions that arise in this evolving field. He discusses the potential impact of AI relationships on traditional monogamy and suggests that societal views and norms will need to adapt to this new aspect of human experience.
The Role of AI Accelerators and Tiny Grad
George Hott emphasizes the significance of decentralized power in the field of AI accelerators. He highlights the challenges faced by companies like Graphcore, Tensort, and others in building AI chips and software ecosystems. Hott asserts that software is the key focus and that Tiny Grad aims to simplify the process of porting new ML accelerators quickly. He also predicts a potential pivot for Tensort towards risk five CPUs as AI accelerators are essentially a software problem rather than a hardware one. He mentions the importance of writing a performance stack for NVIDIA GPUs before considering the development of specialized AI chip hardware.
Decentralized Power and the Future of GPUs
The podcast episode explores the importance of diversifying GPUs and extending decentralization to other areas. The speaker emphasizes the need to reduce centralized power and highlights the potential dangers of monopolistic control in the GPU industry. AMD is mentioned as a potential alternative to NVIDIA, with a focus on performance and stability.
The TinyBox: A Powerful AI Compute Solution
The podcast discusses the creation of the TinyBox, a high-performance deep learning box capable of running large AI models. The TinyBox is designed to be compact, energy-efficient, and easily deployable in a home setting. It features powerful GPUs, high GPU RAM, and fast drive read bandwidth. The speaker highlights the potential of the TinyBox in enabling real-time training and inference for machine learning applications.
Open Source, AI Safety, and the Future of the Internet
The podcast delves into the role of open source in AI development, with a focus on the risks and benefits associated with AI safety. The speaker advocates for open sourcing model architectures and breakthroughs while acknowledging the need to protect proprietary model weights. They express concern about centralized control and promote the idea of decentralized power as a means to combat potential harm. The discussion also touches on the transformative effects of the internet and the importance of preserving its decentralized spirit.
The Importance of Simplifying Code and Refactoring
In the podcast, the speaker emphasizes the significance of simplifying code and prioritizing refactoring. They discuss how the Twitter codebase could benefit from a refactor to reduce complexity and remove unnecessary code. The speaker suggests implementing comprehensive tests and focusing on making the codebase more modular and easier to understand. They argue that a refactor could lead to faster development and improved productivity, though they acknowledge the challenges of refactoring a large-scale and live service like Twitter.
The Role of Technical Leadership and Building Trust in Codebases
The speaker highlights the importance of technical leadership in maintaining high-quality codebases. They emphasize the need for managers and leaders to have deep technical understanding and the ability to recognize skill in software development. They propose a culture of simplicity and continually improving code, prioritizing refactoring over adding new features. Additionally, the speaker suggests integrating comprehensive testing and implementing tools that maximize the quality of the coding experience for individuals. They also discuss the challenges of integrating tests into a codebase with minimal existing tests but highlight the benefits of building trust in code changes and improving development speed.
The Future of AI in Programming
George discusses the potential benefits of using artificial intelligence and machine learning in programming. He expresses his desire for AI-powered tools that can provide feedback and simplify code, while also highlighting the importance of maintaining control and ensuring the alignment of AI with human values and goals.
The Role of AI in Gaming and Virtual Reality
George shares his passion for computer gaming and virtual reality, emphasizing the potential of intelligent AI to enhance gaming experiences. He envisions a future where AI-powered NPCs in games have advanced conversational abilities and can dynamically interact with players, ultimately creating more immersive and captivating virtual worlds.
OUTLINE:
Here’s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time.
(00:00) – Introduction
(08:04) – Time is an illusion
(17:44) – Memes
(20:20) – Eliezer Yudkowsky
(32:45) – Virtual reality
(39:04) – AI friends
(46:29) – tiny corp
(59:50) – NVIDIA vs AMD
(1:02:47) – tinybox
(1:14:56) – Self-driving
(1:29:35) – Programming
(1:37:31) – AI safety
(2:02:29) – Working at Twitter
(2:40:12) – Prompt engineering
(2:46:08) – Video games
(3:02:23) – Andrej Karpathy
(3:12:28) – Meaning of life
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
