> For the complete documentation index, see [llms.txt](https://blog.sunilgudivada.dev/notebook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.sunilgudivada.dev/notebook/transformers-and-llms/transformers.md).

# Transformers

2017 -> "attention you all need" paper

Relies on the self-attention mechanism

It is based on the concept of **Q**uery **K**ey **V**alue

$$
{Attention}(Q,K,V)=\operatorname{softmax}!\left(\frac{QK^\top}{\sqrt{d\_k}}\right)V
$$

**Architecture**

It is based on the encoders and decoders,