Tocanization and Sequencing

i, of course, will report to the show notes of this episode on the official web site, data signs at home dot com. But the tocanization, as as i said, has been applied in a different flavor of tocanizations, to different, to different data types. So, for example, text is and coded with the sentence piece with te something like 32 thousand sub words into the interger. Discreet and continuous values are also tocenized with major order,. Both, in case they are discreet, both, or o floating point values as well. And then each pixl in the image patches is then normalized between arange negative one and one.