> For the complete documentation index, see [llms.txt](https://blog.sunilgudivada.dev/notebook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.sunilgudivada.dev/notebook/transformers-and-llms/tokenization.md).

# Tokenization

Models understand numbers not text, we need to do something that makes the models to understand Natural language text 

Ex:  A Cute teddy bear is reading 

Unit of text is called token

**Arbitrary**: `A`, `cute`, `teddy bear`, `is`, `reading`, `.` 

**Word**:  `A`, `cute`, `teddy`, `bear` , `is`, `reading`, `.` 

**Sub word:** leveraging the roots of words, 

con -> make the sequence longer

 `A`, `cute`, `ted`,`##dy` ,`bear` , `is`, `read`,`##ing` ,`.` 

**Character level tokenizer:** each character is split into token

| Method          | Pros                                                                               | Cons                                                                            |
| --------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Word Level      | <ul><li>Simple</li><li>Interpretable</li></ul>                                     | <ul><li>Risk of OOV</li><li>Does not leverage knowledge of root</li></ul>       |
| Sub word level  | <ul><li>Leverages common prefix and suffix</li><li>Learned from the data</li></ul> | <ul><li>Risk of OOV, though less than word-level</li></ul>                      |
| character level | <ul><li>Small change of OOV</li><li>RoBUsT tO CASinG anD misPeliNgS</li></ul>      | <ul><li>Makes computation slower</li><li>Embeddings not interpretable</li></ul> |