How Does AI Work? (Robert Wright & Timothy Nguyen)
Robert Wright's Nonzero
00:00
The Importance of Bite Pair Encoding in Word Recognition
Chachi B.T. is able to stitch together these sub word units into into word units because when you unroll it it is producing coherent word after word. Even though you're working at this more granular level it still is never the less able to synthesize all of that to get higher level meaning. It's actually producing these sub word tokens and it's predicting those one at a time in practice. The way it's rendered it looks like it's doing word by word but it's actually doing it at the sub word level.
Transcript
Play full episode
