Activation Functions
Why do we need activation functions?
They add non-linearity. Without one, every layer is just a linear map, and stacking linear maps gives… another linear map — so a 100-layer net would be no more powerful than a single layer. The non-linearity is what lets networks model curves, boundaries, and complex patterns.
Sigmoid and tanh squash into a fixed range (and saturate at the ends); ReLU passes positives through unchanged and zeros out negatives.
- “What if you used a linear activation?” — The network collapses to a single linear layer — no benefit from depth.
Sigmoid vs tanh vs ReLU — when to use which?
ReLU is the default for hidden layers — cheap and it doesn't saturate for positive inputs, so gradients flow. Sigmoid and tanh saturate at the ends (flat → tiny gradients), which slows deep training.
Fast, sparse, no positive-side saturation. The default. Variants: Leaky ReLU, GELU, ELU.
Range (−1, 1) centers activations, which helps optimization — but it still saturates.
Range (0, 1) → use for a binary-probability output or gates (LSTM), rarely in hidden layers.
- “Why does saturation hurt?” — Flat regions have near-zero derivative, so gradients vanish through deep stacks. (See Vanishing Gradients.)
- “What's GELU?” — A smooth ReLU-like activation used in transformers (BERT, GPT).
What is the dying-ReLU problem?
If a neuron's inputs always push it negative, ReLU outputs 0 — and its gradient is 0 too, so it never updates again. It's effectively dead. A large learning rate can knock many neurons into this state.
Use Leaky ReLU (small slope for negatives) or ELU/GELU so there's always some gradient, lower the learning rate, and initialize weights sensibly (He init).
- “How does Leaky ReLU help?” — It outputs
0.01·zfor negatives, so the gradient is never exactly zero.
When do you use softmax?
In the output layer for multiclass classification. Softmax turns a vector of scores into probabilities that are positive and sum to 1, so you can read the output as a distribution over classes.
Use sigmoid for binary or multi-label (classes independent); use softmax for multi-class where exactly one class is correct. Pair softmax with cross-entropy loss.
- “Why exponentiate?” — It keeps everything positive and amplifies the largest score while preserving a smooth distribution.
- “What is temperature in softmax?” — Dividing logits by T sharpens (low T) or flattens (high T) the distribution. (See Sampling & Decoding.)