Tokenization

Models understand numbers not text, we need to do something that makes the models to understand Natural language text

Ex: A Cute teddy bear is reading

Unit of text is called token

Arbitrary: A, cute, teddy bear, is, reading, .

Word: A, cute, teddy, bear , is, reading, .

Sub word: leveraging the roots of words,

con -> make the sequence longer

A, cute, ted,##dy ,bear , is, read,##ing ,.

Character level tokenizer: each character is split into token

Method	Pros	Cons
Word Level	Simple Interpretable	Risk of OOV Does not leverage knowledge of root
Sub word level	Leverages common prefix and suffix Learned from the data	Risk of OOV, though less than word-level
character level	Small change of OOV RoBUsT tO CASinG anD misPeliNgS	Makes computation slower Embeddings not interpretable