Transformers

Cross-source synthesis of the Transformer architecture and its evolution. Updated as new research is ingested.

Overview

The Transformer (Vaswani et al., 2017) replaced recurrence with self-attention, enabling parallel processing of entire sequences. Every major modern LLM — GPT, Claude, Gemini, Qwen — is built on this foundation.

"Attention is all you need." — The paper that changed AI.

Architecture

Core mechanism: self-attention

Attention lets each token attend to every other token in the sequence simultaneously. For each word, the model computes:

• Query (Q) — what am I looking for?
• Key (K) — what do I contain?
• Value (V) — what information do I pass on?

Attention weights determine how much each token influences every other token. This solves the long-range dependency problem that plagued RNNs.

See Attention Is All You Need Explained for the full beginner-friendly explainer.

Encoder vs decoder

The original Transformer has two halves:

Half	Role	Used by
Encoder	Understand input bidirectionally	BERT, encoder-only models
Decoder	Generate output left-to-right	GPT family, decoder-only LLMs

Modern LLMs are overwhelmingly decoder-only — they predict the next token autoregressively.

Evolution timeline

2017 — Foundation

Attention Is All You Need Explained — Introduced multi-head attention, positional encoding, and the encoder-decoder architecture. Originally designed for machine translation.

2018 — Understanding language

Bert Pre Training Transformers Explained — BERT uses the encoder half with bidirectional pre-training (masked language modeling). One model, many downstream tasks via fine-tuning.

2019 — Generating language

Gpt2 Language Models Unsupervised Explained — GPT-2 uses the decoder half, trained on next-token prediction. Showed that scale produces emergent fluency.

2020 — Scale changes everything

Gpt3 Few Shot Learners Explained — GPT-3 (175B parameters) demonstrated in-context learning: perform new tasks from examples in the prompt, no fine-tuning needed.

Scaling Laws Compute Optimal Explained — Kaplan et al. showed predictable power-law relationships between model size, data, and compute. Bigger models + more data = better performance.

2021–2022 — Alignment and instruction following

Instructgpt Rlhf Explained — RLHF aligns models to human preferences. The bridge from raw language modeling to helpful assistants.

Flan Instruction Tuning Explained — Instruction tuning at scale: fine-tune on diverse tasks phrased as instructions. Cheaper than RLHF, nearly as effective.

Chain Of Thought Reasoning Explained — Prompting models to "think step by step" unlocks reasoning capabilities without architectural changes.

Modern implications

Sparse architectures

Mixture Of Experts Sparse Models Explained — MoE routes each token to a subset of expert layers, achieving large effective parameter counts without proportional compute cost. Used in Mixtral, Qwen-MoE, and others.

Inference optimization

Modern deployment stacks layer multiple techniques on top of the Transformer base:

• Quantization (INT8, FP8, INT4)
• KV-cache optimization
• Speculative decoding

See Inference Optimization Quantization Sparsity Speculative Decoding 2026 05 12 for the 2026 landscape.

Key papers (source summaries)

Paper	Year	Summary page
Attention Is All You Need	2017	Attention Is All You Need Explained
BERT	2018	Bert Pre Training Transformers Explained
GPT-2	2019	Gpt2 Language Models Unsupervised Explained
GPT-3	2020	Gpt3 Few Shot Learners Explained
Scaling Laws	2020	Scaling Laws Compute Optimal Explained
InstructGPT / RLHF	2022	Instructgpt Rlhf Explained
FLAN	2022	Flan Instruction Tuning Explained
Chain-of-Thought	2022	Chain Of Thought Reasoning Explained

Companion code

Hands-on Python demos for several papers: Ai Papers Explained Python Demos and Ai Papers Explained Python Demos Part 2.

Open questions

• Efficiency ceiling — Can sparse/MoE architectures match dense models at equal compute?
• Reasoning — Does chain-of-thought scale to complex multi-step problems, or do we need architectural changes?
• Context length — Attention is O(n²) in sequence length; long-context models rely on approximations (sliding window, ring attention, etc.)

Related concepts

• Mixture Of Experts Sparse Models Explained — Sparse Transformer variants
• Open Source Llm Deployment Architectures 2026 — Deploying Transformer models in production