AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
3min chapter
#306 – Oriol Vinyals: Deep Learning and Artificial General Intelligence
Lex Fridman Podcast
The Concept of Tokenization in Machine Learning
The approach we usually do in machine learning when we deal with images and we do this quantization step is a bit more data driven so rather than have some sort of Fourier basis for how you know frequencies appear in natural. We actually just use the statistics of the images and then quantize them based on the statistics much like you doing words right? So common substrings are located a token an image is very similar but there's no connection the token space if you think of all like the tokens are an integer and in the end of the day that makes it orthogonal. The model will need to predict the tokens from the text green field to then the pixels and that will start making the
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode