How Open-Source LLMs Are Built

The big picture

Almost every open-source LLM you've heard of — Llama, Mistral, Qwen, DeepSeek, Gemma, gpt-oss, Phi — is built from the same handful of LEGO bricks. They all stack the same kind of layer dozens of times. What makes them different is which version of each brick they pick.

This page is the map. First we walk the shared skeleton — the decoder-only transformer — one layer at a time, with an animation for each brick. Then a side-by-side table shows exactly which brick each model chose. Finally, pick any model for its own deep-dive.

One shared recipe

A modern open LLM is a tall sandwich. Text comes in at the bottom, gets turned into numbers, flows up through N identical blocks (often 28–80 of them), and a prediction for the next word comes out the top. Every block does two jobs: let tokens talk to each other (attention) and think about each token on its own (a feed-forward network). Normalization and "shortcut" (residual) connections wrap both jobs so the tall stack stays trainable.

The animation below walks that whole journey. Press Play, or step through with Next.

Brick 1 — RoPE: how the model knows word order

Attention by itself is order-blind: shuffle the words and it can't tell. Modern open models fix this with RoPE (Rotary Position Embedding). Instead of adding a position signal, RoPE rotates each token's query and key vectors by an angle that grows with its position. The clever part: when two tokens compare, only the angle between them matters — so the model learns relative distance, and it stretches to longer texts more gracefully.

Drag the slider to move the whole sentence later in the document. Watch the vectors spin — but the gap between any two tokens stays fixed. That fixed gap is what attention actually reads.

Brick 2 — RMSNorm & pre-norm

Deep stacks are fragile: numbers can blow up or vanish as they pass through dozens of layers. Normalization rescales each token's vector to a sane size before each sub-layer. Modern models use RMSNorm — a cheaper, simpler cousin of LayerNorm that just divides by the vector's root-mean-square magnitude (no mean-subtraction). They also normalize before each sub-layer ("pre-norm") rather than after, which makes very deep models far easier to train.

Brick 3 — SwiGLU: the feed-forward network

After tokens mix via attention, each token is processed alone by a small two-layer network — the feed-forward network (FFN), where most of a dense model's parameters live. Modern models replace the plain FFN with a gated one: SwiGLU (and Gemma's close relative GeGLU). A gate lets the network decide, per dimension, how much signal to let through — a small change that reliably improves quality.

Brick 4 — the attention family: MHA → MQA → GQA → MLA

Attention needs to remember a Key and Value for every past token — the KV cache. On long chats that cache becomes the memory bottleneck. The big efficiency story of open LLMs is shrinking it. Each step keeps the same number of query heads but shares or compresses the keys and values.

Click each option to see the keys and values collapse — and the KV-cache bar shrink with them.

Brick 5 — Mixture-of-Experts (MoE)

How do you make a model "know more" without making every token cost more to compute? Mixture-of-Experts. Replace the single FFN with many expert FFNs, and add a tiny router that sends each token to just a few of them. The model has huge total parameters, but only a small active slice runs per token. Mixtral, DeepSeek-V3, gpt-oss, Qwen3-MoE and Llama 4 all use this.

Watch one token get routed: the gate scores all experts, the top few light up, they process, and their outputs are blended.

Brick 6 — long context without quadratic cost

Full attention compares every token to every other — cost grows with the square of the length. To read long documents cheaply, models limit who each token may look at. Sliding-window attention (Mistral) lets a token see only the last W tokens. Local + global layers (Gemma, gpt-oss) alternate cheap local layers with occasional full-range ones. Cost grows roughly linearly instead.

Flip between mask types and drag the window. Lit cells = "this query (row) is allowed to look at this key (column)."

Brick 7 — tokenizer & vocabulary

Before any of this, text must become tokens. Models use byte-level BPE or SentencePiece to split text into sub-word pieces. A bigger vocabulary means fewer tokens per sentence (cheaper, longer effective context) but a larger embedding table. Vocabularies have grown a lot: Llama 2 used 32k, Llama 3 jumped to 128k, and Gemma uses 256k — great for multilingual and code text.

Same bricks, different choices

Here's the whole field at a glance. Every model is a decoder-only transformer with RoPE and RMSNorm — the differences are in attention, the FFN, and whether they go Mixture-of-Experts. Click a row to highlight it and open that model's deep-dive.

Pick a model

Each deep-dive shows the family's timeline, its signature architecture trick with its own animation, and how it differs from the shared recipe.