> For the complete documentation index, see [llms.txt](https://blog.sunilgudivada.dev/notebook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.sunilgudivada.dev/notebook/transformers-and-llms/token-represenation.md).

# Token Represenation

One hot encoding -> vector representation of 0, 1

Learned embedding

### **Word2Vec** 

Neural network with a proxy task over billions of words worth of text

learns an embedding layer

**Proxy tasks:** 

Continuous bag of words 

`A` `Cute` teddy bear `is`  `reading`

 Skip Gram

A Cute `teddy bear` is reading

Cross Entrophy -> it will determine how far off you are from the true answer

epoch -> number of times model sees the training set