RNNs, LSTM & GRU
What is an RNN and what does it solve?
A recurrent network processes a sequence one step at a time, carrying a hidden state forward as memory. The same weights are reused at every step, so it handles variable-length inputs like text, audio, or time series.
The same cell runs at each time step, passing its hidden state (green) forward. That state is the network's memory of everything seen so far.
- “What's a bidirectional RNN?” — Two RNNs, one reading forward and one backward, so each step sees both past and future context.
Why do RNNs struggle with long sequences?
Because the same weight is multiplied at every step, gradients across a long sequence either vanish or explode — so a plain RNN can't connect information that's many steps apart (long-range dependencies).
- “Concrete example?” — “The cat, which … , was hungry” — matching the verb to a far-back subject needs long memory.
- “Root cause?” — The repeated multiplication of derivatives. (See Vanishing & Exploding Gradients.)
How do LSTMs fix it?
An LSTM adds a cell state — a memory highway that information can travel along almost unchanged — controlled by gates that learn what to keep, forget, and output. Gradients flow along the cell state without vanishing.
Decides which parts of the old memory to erase.
Decides which new information to write into the cell state.
Decides what of the cell state becomes the visible hidden state.
- “Why does the cell state help gradients?” — It's updated mostly by addition, not repeated multiplication, so gradients stay healthy across time.
GRU vs LSTM — which to use?
A GRU is a streamlined LSTM: two gates instead of three, no separate cell state — fewer parameters, faster, and often just as accurate. Try GRU first; reach for LSTM if you need more capacity.
- 2 gates (reset, update); merges cell & hidden state.
- Fewer params → faster, good on smaller data.
- 3 gates + separate cell state.
- More expressive on long, complex sequences.
- “Are RNNs still used?” — Largely replaced by Transformers for NLP, but still useful for streaming / low-latency sequence tasks.