Mixture of Experts
What is a Mixture of Experts?
MoE replaces a transformer's dense feed-forward layer with many parallel “expert” sub-networks plus a router that sends each token to only a few of them. The model has a huge total parameter count, but only a small fraction runs per token.
- “Where in the transformer?” — Typically the feed-forward block of each transformer layer becomes an MoE layer; attention stays dense.
How does the router work?
A small learned gating network scores the experts for each token and picks the top-k (often top-1 or top-2). Only those experts compute; their outputs are combined, weighted by the gate scores.
This is “conditional computation”: different tokens take different paths through the network, so capacity grows without every token paying for every parameter.
Total vs active parameters — what's the point?
MoE decouples capacity from cost. A model can hold, say, hundreds of billions of total parameters but only activate a dozen billion per token, so it has the knowledge of a giant model at the inference cost of a much smaller one.
All experts combined — sets how much the model can store.
Only the routed experts run — sets the FLOPs per token.
Every expert sits in memory even if rarely used — a real cost.
What is load balancing and why does it matter?
Left alone, the router can collapse onto a few favorite experts — others starve and waste capacity. A load-balancing (auxiliary) loss encourages the router to spread tokens evenly across experts.
- “What's expert capacity?” — A cap on tokens per expert per batch; overflow tokens get dropped or passed through, which is why balancing matters for both quality and efficiency.
MoE vs a dense model — the trade-offs?
MoE buys efficiency at the cost of complexity.
- More capacity per FLOP — strong quality at lower inference compute.
- Faster to train to a given quality than an equally-capable dense model.
- High memory (all experts loaded) and complex distributed serving.
- Training instability and routing/balancing headaches.