Transformers & Attention
What is self-attention?
Self-attention lets every token look at every other token and pull in the ones that matter. Each word's new representation is a weighted blend of all words, where the weights come from how relevant each is — so “it” can attend to the noun it refers to.
The query “it” computes a relevance score with every word; the thickest link (to “animal”) is the highest attention weight. The output is the weighted sum.
- “Why divide by √dₖ?” — Scaling the dot products keeps them from getting large and pushing softmax into tiny-gradient regions.
What are query, key, and value?
Each token is projected into three vectors. The query asks “what am I looking for?”, each key advertises “what I offer”, and the value is the content returned. Attention = softmax(Q·Kᵀ/√dₖ)·V.
Compared against every key to score relevance.
The match target; Q·K gives the attention scores.
The actual content blended together using the softmax weights.
- “Cross-attention vs self-attention?” — Self: Q, K, V from the same sequence. Cross: Q from the decoder, K/V from the encoder (used in translation).
Why multi-head attention?
One attention head captures one kind of relationship. Multiple heads run in parallel, each learning a different pattern — syntax, coreference, position — and their outputs are concatenated. It's like several attention “experts” at once.
- “Does it cost more compute?” — Not much — each head works in a smaller subspace, so total size is similar to one big head.
Why did transformers replace RNNs (and why do they need positional encoding)?
Transformers process all tokens in parallel (not step-by-step like RNNs), so they train far faster and capture long-range dependencies directly via attention. But parallelism means they have no inherent sense of order — so positional encodings are added to give each token its place.
- Sequential — can't parallelize across time.
- Long-range memory is hard (vanishing gradients).
- Fully parallel; every token attends to every token directly.
- Scales to huge models — the backbone of LLMs.
- “Encoder vs decoder?” — BERT is encoder-only (understanding); GPT is decoder-only (generation); T5 uses both.
- “What's the cost of attention?” — O(n²) in sequence length — the motivation behind efficient-attention variants and limited context windows.