Seq2Seq & Attention
What is a seq2seq (encoder-decoder) model?
It maps one sequence to another of possibly different length. An encoder reads the input into a representation; a decoder generates the output token by token from it. Perfect for translation, summarization, and Q&A.
The encoder compresses the source; the decoder generates the target one token at a time, conditioned on the encoding.
- “What did early seq2seq use?” — Stacked LSTMs for both encoder and decoder.
What was the bottleneck problem?
Cramming an entire input sentence into a single fixed-size context vector loses information — especially for long sentences. The decoder only sees that one summary, so quality drops as length grows.
- “Why long sentences specifically?” — A fixed vector can't hold more detail as the input grows, so early words get “forgotten.”
How did attention fix it?
Attention lets the decoder look back at all encoder states at every output step, focusing on the most relevant source words — no single bottleneck vector. This was the breakthrough that led to the Transformer.
“Attention is all you need” dropped recurrence entirely and used only attention — enabling full parallelism and the modern LLM era.
- “Attention here vs self-attention?” — This is cross-attention (decoder→encoder); transformers also use self-attention within each sequence.
What tasks does it power?
Any sequence-to-sequence mapping: machine translation, summarization, question answering, code generation, and speech-to-text.