Transformer

The transformer architecture, introduced in the 2017 paper "Attention Is All You Need", revolutionised AI by enabling efficient processing of sequences without the limitations of recurrent networks.

Key innovations of transformers:

• Attention mechanism: Allows direct connections between any positions in a sequence
• Parallel processing: Unlike RNNs, processes all positions simultaneously
• Positional encoding: Injects position information into the parallel architecture
• Layer normalisation: Stabilises training of deep networks

Transformer components:

• Self-attention layers: Each position attends to all others
• Feed-forward layers: Process each position independently
• Residual connections: Help train very deep networks
• Multi-head attention: Multiple parallel attention operations

Why transformers won:

• Scale better with compute and data
• Train faster (parallelisable)
• Handle long-range dependencies
• Emerge with new capabilities at scale

Transformer variants:

• Encoder-only (BERT): Understanding tasks
• Decoder-only (GPT): Generation tasks
• Encoder-decoder (T5): Sequence-to-sequence tasks