Transformers

2017 -> "attention you all need" paper

Relies on the self-attention mechanism

It is based on the concept of Query Key Value

$${Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Architecture

It is based on the encoders and decoders,