For the complete documentation index, see llms.txt. This page is also available as Markdown.

Token Represenation

One hot encoding -> vector representation of 0, 1

Learned embedding

Word2Vec

Neural network with a proxy task over billions of words worth of text

learns an embedding layer

Proxy tasks:

Continuous bag of words

A Cute teddy bear is reading

Skip Gram

A Cute teddy bear is reading

Cross Entrophy -> it will determine how far off you are from the true answer

epoch -> number of times model sees the training set

Last updated