Tokenization: How AI Reads Text

Tokenization: How AI Reads Text

AI doesn't read words. It reads bricks of text called tokens.