Sparse Everywhere

The cheapest way to make a model more capable is no longer to make all of it bigger. It is to make more of it, and run less of it.

Three releases in one week in October showed the same bet at three very different sizes. Reflection's Beam holds 501 billion parameters and runs 23 billion for each token [10]. Mistral Large 4 holds about 1.05 trillion and runs 52 billion [11]. And Google's EmbeddingGemma 2, at 740 million parameters, is built as a 270 million parameter text core with separate vision and audio encoders that its card calls "selectively loadable", so a developer can "load only the modalities required for their use case" [12]. One model is a thousand times larger than another, and they are built on the same idea: hold more than you run, and let the input decide what runs.

That idea has a name inside the model, mixture-of-experts, where a router picks a few specialist blocks for each token and leaves the rest idle. It has no single name at the other scales, but it is the same shape, and this article is about the shape rather than the week. The companion pieces cover what the two sizes cost you and how much of the October evidence you can trust. This one asks why every serious lab now makes the same choice.

Capability is now cheaper to route than to grow. Hold more than you run, and let the input decide what runs. That is one principle, and it is showing up at every altitude of the stack.

Where the bet came from

The idea is older than the transformer. In 1991 Jacobs, Jordan, Nowlan and Hinton trained a set of small networks with a gating network that decided which one should handle each input, and called it a mixture of local experts [1]. It stayed a curiosity until 2017, when Shazeer and colleagues put a sparsely gated expert layer inside a language model, reached 137 billion parameters, and reported capacity gains of up to a thousand times "with only minor losses in computational efficiency" [2]. GShard scaled the recipe to 600 billion in 2020 [3], and the Switch Transformer showed in 2021 that at the same compute budget the sparse model pretrains up to seven times faster [4].

The open-weight turn came in January 2024, when Mixtral 8x7B matched or beat Llama 2 70B while running 13 billion of its 47 billion parameters per token [5]. DeepSeek-V3 then made the modern proportions familiar: 671 billion total, 37 billion active, trained in 2.79 million H800 GPU hours [6]. Through 2025 the total count kept climbing while the active count stayed put: Llama 4 Maverick at 400 billion with 17 active [9], Kimi K2 at a trillion with 32 active [8], and OpenAI's gpt-oss-120b at 117 billion with 5 active [7]. Beam and Mistral Large 4 sit on the same line.

Why routing wins

Two numbers explain the chart. Compute per token scales with the active count, and so does most of the training cost, which is the result the Switch Transformer established [4]. A lab that adds experts adds capacity at a price that tracks the small number, not the big one. The big number costs memory, and memory is paid for once and shared across every request on the machine. At the scale a frontier lab serves, that trade is lopsided in favour of holding more.

There is a second reason that is easier to miss. A dense model spends the same parameters on every token, whether the token is a comma or a line of Rust. A sparse model can send the comma through cheap experts and the Rust through specialist ones. The model is bigger for the same per-token cost, and it spends that cost where the input is hard. Routing is a form of attention paid in parameters instead of tokens, and once you see it that way the pattern stops being a trick of large language models and starts being a general design rule.

Sparse at three altitudes

Look at the three October releases through that rule and they line up as one principle at three heights in the stack.

Inside the model, the experts are held and the router runs a few of them for each token. That is Beam and Mistral Large 4, and every model in Figure 1. At the edge of the model, EmbeddingGemma 2 holds a text core plus vision and audio encoders, and the developer loads only the encoders the workload needs; its card also supports truncating each embedding from 768 to 128 dimensions when the search does not need the precision [12]. The decision is made per modality and per deployment rather than per token, but the shape is the same: more is held than is run. Above the model, a routing layer holds a fleet of models and sends each request to the one that fits, a pattern this series has looked at from two sides: "From Router to Conductor" on what the chooser should be, and "Routing Above the Model" on why the harness around the chooser moves cost further than the choice of model does. There the decision is made per request. At every altitude the same two questions decide the design: what is held, and who chooses what runs.

What it changes for you

If routing is the cheapest way to add capability, then the constraint moves. For a decade the question was how much compute you could afford. The question now is how much you can hold, and how busy you can keep it. Memory is the price of a sparse model, and utilization is the discount; the companion article on the active parameter works through that arithmetic. A team that buys a sparse model and runs it at a third of capacity has bought the big number and received the small one.

The second change is to your own designs. The rule that made these models cheap applies to any system you build on them. Hold the tools, the documents, the specialist models and the prompts you might need. Run only what the request needs. And be explicit about who chooses, because the chooser is where the quality lives: a router that sends everything to the biggest model has recreated a dense system at sparse prices, and a router that sends too much to the smallest has recreated a cheap system with expensive failures. Google made the chooser a configuration flag. Reflection and Mistral made it a learned layer. Your agents will need one too, and it deserves the same care as the models it chooses between.

Conclusion

The 1991 paper's title was "Adaptive Mixtures of Local Experts", and thirty-five years later that is still the most accurate description of where the field has arrived: not one giant mind, but a large collection of local competence and a mechanism for picking from it. The trillion-parameter model and the 740 million parameter one are the same idea with a different chooser.

Expect the total counts to keep climbing and the active counts to stay roughly where they are, because that is the shape the economics reward. Expect the same shape to appear in your inference stack, your retrieval layer and your agent fleet. The labs have already decided that capability is cheaper to route than to grow. The remaining question is whether the systems built on top of their models will make the same decision deliberately, or by accident.

What leaders should do

1. Change the sizing question in every model review from "how much compute" to "how much memory, at what utilization". Sparse models move the bottleneck, and a budget built for dense models will buy the wrong hardware.
2. Audit your own stack for dense spending: any step that sends every request through the largest model, loads every encoder, or retrieves every document. Each one is a place where a chooser would cut cost without cutting quality.
3. Treat the chooser as a product. Give the routing layer an owner, a measured error rate, and an evaluation set, because its mistakes are invisible in the model's benchmark scores and visible in your bill.
4. Prefer components that ship modular, the way EmbeddingGemma 2 ships its encoders. A model you can load in parts is a model whose cost you can shape to the workload instead of the other way around.