Neural Network Fundamentals
What does a single neuron compute?
A neuron takes a weighted sum of its inputs, adds a bias, and passes the result through a non-linear activation: a = f(w·x + b). The weights say how much each input matters; the activation lets it model non-linear patterns.
Learned importance of each input — what the network tunes during training.
Shifts the activation threshold so the neuron can fire even when inputs are zero.
- “What's a perceptron?” — The simplest neuron with a step activation; it can only separate linearly-separable data.
Why do we need hidden layers and depth?
Stacking layers lets the network build features of features: early layers learn simple patterns (edges), later layers combine them into complex ones (shapes, objects). Depth makes it exponentially more efficient than a single wide layer.
A single hidden layer is a universal approximator in theory, but it may need impractically many neurons. Depth reuses learned features hierarchically, so deep nets generalize better with fewer parameters.
- “What is the universal approximation theorem?” — One hidden layer with enough neurons can approximate any continuous function — existence, not efficiency.
- “What makes a network 'deep'?” — Multiple hidden layers; 'deep learning' is just neural nets with many layers.
What is forward propagation?
Forward propagation is the prediction pass: feed inputs through each layer in turn, computing a = f(W·x + b) at every layer until the output. It's just a chain of matrix multiplies and activations.
- “Why use matrices?” — A whole layer is one matrix multiply, which GPUs run in parallel across the batch.
How does a neural network learn?
It loops: forward pass to predict, a loss to measure error, backpropagation to compute how each weight affected the loss, then gradient descent to nudge weights downhill. Repeat over many batches.
Run inputs through the network to get an output.
Propagate the error backward to get each weight's gradient.
- “What is an epoch?” — One full pass over the training data; training runs many epochs.
- “Where do the weights start?” — Randomly initialized (e.g. Xavier/He) — never all zeros, or every neuron would learn the same thing.
- “What's the role of backprop here?” — It's the efficient chain-rule algorithm that gives every weight its gradient in one backward sweep. (See Backpropagation.)