S9:E4 - "Confused about compilers?"
Base.cs Podcast
00:00
How to Tokenize a Programming Language
Once you have all the data recorded, effectively what you can do is transform this big chunk of text that we started off with into substrings. And now you're like, okay, now I can start to see individual words inside of this. These words, they have a name called Lexsemes. Oh, so those are the individual substrings that you get once you're done scanning. So if you had a scanner taking care of the scanning, the Lexsere is going to take over and sort of determine, oh, okay, this Lexseme, this word is this token and it's going to sort of classify everything.
Transcript
Play full episode
