If you've wondered how models keep getting dramatically larger without inference becoming impossibly expensive, the answer for most of the frontier is a single architectural idea: mixture of experts, or MoE. It's one of the most important concepts in modern LLMs and it's genuinely not that hard to understand.

The core idea

A standard ("dense") transformer runs every parameter for every token. If the model has 70 billion parameters, all 70 billion participate in processing each token. That's why bigger dense models cost proportionally more to run — capacity and compute are locked together.

MoE breaks that link. Instead of one big feed-forward network in each layer, an MoE layer has many smaller ones called experts — say 8, 16, or many more. A small router network looks at each token and picks just a couple of experts to handle it. The rest sit idle for that token.

The consequence is the headline feature: a model can have, say, hundreds of billions of total parameters but only activate a small fraction — the "active parameters" — for any given token. You get the knowledge capacity of a huge model with the compute cost of a much smaller one.

Why this matters

Capacity without proportional cost. More total parameters generally means more knowledge and capability. MoE lets you scale total parameters aggressively while keeping the per-token compute — and therefore the inference cost and latency — far lower than a dense model of the same total size.

Specialisation. Different experts can, in effect, specialise. The router learns to send code-like tokens to experts that handle code well and prose to experts that handle prose. Nobody hand-assigns these specialities; they emerge during training.

It's why the frontier is servable. Many of the largest, most capable models would be prohibitively expensive to run if every parameter fired on every token. MoE is a big part of how labs ship models with enormous capacity that still respond quickly enough to be useful.

The catch: memory

MoE saves compute, not memory. Even though only a few experts run per token, you don't know in advance which ones the router will pick — and across a batch of tokens, all of them get used. So every expert has to be loaded into memory and ready. A model with 8x the total parameters of a dense one needs roughly 8x the memory to hold, even if it only does a fraction more compute per token.

This is the central trade-off. MoE shifts the bottleneck from compute to memory and memory bandwidth. It's a great deal when you have the RAM but want to keep compute and latency down — which describes most serving setups — but it's not a free lunch.

Training challenges

MoE is harder to train than a dense model. Two problems recur:

Load balancing. If the router sends most tokens to a few favourite experts, the others never learn and capacity is wasted. Training includes special objectives to encourage the router to spread tokens evenly across experts.

Routing instability. Small changes in routing can cause large changes in behaviour, which can make training less stable. A lot of the engineering in MoE is about keeping the router well-behaved.

What it means for builders

For most people consuming models through an API, MoE is invisible plumbing — you just benefit from better capability per pound. But if you're self-hosting, it matters directly: an MoE model may have modest compute needs yet demand a lot of memory to hold all those experts. Size your hardware for total parameters (memory), not active parameters (compute).

MoE isn't the only way forward, and there's active research into making it more efficient and into alternative architectures entirely. But for now it's the workhorse trick behind a lot of the frontier — the reason models can be enormous and still answer you in a couple of seconds.