Transfer Learning
What is transfer learning and why does it work?
Transfer learning reuses a model pretrained on a large dataset as the starting point for a related task with little data. It works because early layers learn general features — edges, textures, or basic language structure — that transfer across tasks.
- “When is it most valuable?” — When your labeled dataset is small but a big pretrained model exists for a similar domain.
Feature extraction vs fine-tuning?
Two ways to reuse the pretrained network, trading compute and data for adaptation.
- Freeze the backbone; train only a new head.
- Fast, little data, low overfitting risk.
- Backbone features stay generic.
- Unfreeze some/all layers and keep training at a low learning rate.
- Adapts features to your domain — higher accuracy.
- Needs more data and care to avoid overfitting.
Which layers do you freeze, and why?
Generally freeze the early layers and adapt the later ones. Early layers capture universal low-level features; later layers are task-specific and benefit most from retraining.
Start with the backbone frozen and train the head; then unfreeze the top layers and fine-tune at a small learning rate. The more similar your data is to the pretraining data, the more you can freeze.
How does it relate to pretraining LLMs?
It's the same idea at scale: pretrain a transformer on huge unlabeled text (self-supervised), then adapt it. Fine-tuning, instruction tuning, and parameter-efficient methods like LoRA are all transfer learning.
- “Why is pretraining self-supervised?” — Predicting masked or next tokens needs no labels, so it can use enormous corpora.
What are the pitfalls?
Transfer isn't free — a mismatch between source and target domains can hurt.
If the domains differ too much, pretrained features can mislead.
Too-high a learning rate erases useful pretrained weights.