I forced a Mixture of Experts to specialize by topic. It got worse.

Every explanation of Mixture of Experts I have read describes a committee of specialists. One expert handles code, another handles medicine, another handles legal text, and a router reads each word and sends it to the right desk. It is a good story. It makes the architecture sound like an office.

I trained five language models from scratch on two consumer GPUs to check it, and the story is wrong in a specific and interesting way. In my models, knowing which expert a word was sent to tells you noticeably more about whether that word was a comma than about whether the document was a Python file or a novel. Worse, that arrangement was fixed within the first 2% of training and never moved again, while the model itself got five times better at its job.

Then I spent about sixty GPU-hours trying to force the committee-of-specialists arrangement into existence. It can be forced. It does not help.

This is the follow-up to an earlier study where I read a pretrained router's decisions with a Jacobian lens and found the same preference for token form. That result left an obvious objection open: maybe the pretraining data caused it. Training from random initialization closes that door.

https://video.wixstatic.com/cf00a8_0b6e1a694b0d49959253fec5a2e5553b

The whole experiment in 90 seconds, narrated. Every number on screen is measured, including all 54 points of the curve.

Where a Mixture of Experts puts its experts

A transformer is a stack of identical blocks. Mine had twelve. Each block does two things to every word, in order: it adds the result of attention, then it adds the result of an MLP.

Attention is the part that looks at other words. When the model processes "sat" in "the cat sat," attention is what lets that position pull information from "cat." The MLP is the opposite: it processes each word on its own, with no knowledge of any other word in the sentence. Roughly, attention moves information between words and the MLP does the thinking about each word individually.

Mixture of Experts replaces the MLP only. Attention is left completely alone, in every block. So instead of one MLP per block you get eight, plus a small router that picks two of them for each word. In my models six of the twelve blocks were built this way.

The reason it is the MLP and not attention is unglamorous. The MLP is where the parameters are. Each of mine held 4.72 million weights against attention's 1.57 million. Eight copies across six blocks came to 226.6 million of the model's 311.5 million weights. Each word only used two of the eight, so the model stored 311.5 million parameters and spent 141.6 million on any given word. That gap is the entire trick: the memory of a large model at the compute cost of a small one.

The router decides per word, not per document

The router does not see the document. It does not see the sentence. It sees one word's vector and nothing else. Each training step processed 8 documents of 1024 words, so the router made 8,192 independent decisions per block, six blocks deep. Inside a single Wikipedia article, "The" might go to experts 3 and 5 while the very next word goes to 1 and 7.

Routing whole documents instead would have broken training outright. Each step would have produced 8 routing decisions rather than 8,192, spread across 8 experts. Most experts would receive nothing on most steps, get no gradient, and die. Per-word routing is not a refinement, it is what makes the thing trainable at all.

Two numbers, and what they mean

Perplexity is how surprised the model is by real text. The useful intuition is: how many equally-plausible options is the model choosing between at each word? My vocabulary had 49,152 tokens, so a model that knew nothing would score about 49,152. After 50 million words mine were at 72.7. At the end they were at 14.4. Lower is better.

Normalized mutual information answers a different question: if I tell you X, how much do you learn about Y? Zero means nothing at all. One means X tells you Y exactly. I measured it twice. Domain NMI asks how much the expert choice tells you about which of five sources the document came from. Form NMI asks how much it tells you about the surface shape of the word: whitespace, punctuation, numbers, alphabetic words, other.

If the committee-of-specialists picture were right, domain NMI would be high and form NMI would be low.

It came out the other way, and then it stopped moving

At 50 million words: perplexity 72.718, domain NMI 0.1826, form NMI 0.2686. At 2.7 billion words: perplexity 14.402, domain NMI 0.1845, form NMI 0.2644.

Perplexity fell by 80%. Domain NMI moved by 0.002. Form NMI moved by 0.004, in the wrong direction. Across 54 measurements the ratio of form to domain never left the range 1.43 to 1.59.

Routing carries about 45% more information about what a word looks like than about what the document is about. That matches what the Mixtral team reported and what later work on part-of-speech sensitivity found. Doing it from random initialization rules out one alternative explanation: nobody can argue the pattern was inherited from a particular pretraining corpus, because there was no pretraining.

The freeze is the part I have not seen stated anywhere. By 50 million words, 2% of the run, the router had picked its partition and held it while the model underneath became unrecognizably better. Expert specialization did not emerge slowly over training. It never started.

I have a guess about why. All eight experts begin as exact copies of each other, so at step one the only thing distinguishing them is the router's random initialization. Whichever words the untrained router happens to send to expert 3, expert 3 gets gradient on, becomes better at them, and the router keeps sending them. The partition is an accident that then makes itself true.

Sparsity is worth a lot, whatever the experts are sorting

Three of the five models were a matched-compute set: a plain model, a dense model with a double-width feedforward layer, and the Mixture of Experts. The dense model and the MoE do exactly the same arithmetic per word. The MoE just has a bigger library to choose from.

The MoE finished at 14.4025 against the matched-compute dense model's 15.5365, a 7.30% win, using 2.2 times the stored parameters. That gap held between 5.35% and 7.46% across 45 matched measurements and grew, rather than closed, through the learning-rate decay. So the experts are doing real work. They are simply not doing the work the folk model claims.

Three ways to force the issue

I tried three ways of insisting, each an independent run of 2.7 billion words against a control sharing its architecture, random seed, data, and data ordering.

The first never showed the router a label. I added a term to the loss that penalized the router when the words of a single domain spread evenly across experts. The label affected only the loss, so at inference the model was an ordinary MoE.

The second and third handed the label to the router directly. There is a wrinkle worth knowing: a linear router cannot do much with a topic input. Whether you concatenate a domain embedding or add one to the hidden state, both reduce algebraically to a per-domain nudge to the scores and nothing more. So I built the honest version of that ceiling, and also a version that escapes it by letting the domain scale and shift the vector the router reads.

Both started at exactly zero. Because torch.zeros draws no random numbers, I could add the new parameters without disturbing the initialization of anything else, and I checked it: every base weight was bit-identical to the control, and the untrained model produced a loss of 10.94829369 with and without the addition, to the last digit.

Forcing works. Pushing harder destroys it.

At the mild setting the router's domain information doubled and the form-to-domain ratio flipped from 1.43 to 0.70. Topic routing turned out to be easy to create.

The mechanism shows up in the form NMI column. At the mild setting form NMI did not move, which means the model added topic structure on top of the surface-form structure it already had rather than trading one for the other. One step stronger, form NMI halved, and that is exactly where the perplexity cost appeared. The damage came from displacing surface-form routing, not from topic routing itself. Whatever the router was doing with word shape, it was load-bearing.

The strongest setting is the one I did not expect. Pushing ten times harder did not produce ten times the topic routing. It produced total collapse: the model abandoned most of its experts, load balance fell from 0.969 to 0.751, and domain NMI dropped to 0.0140, well below where it started.

Over the full runs the mild setting cost 0.87% on average and was behind the control at 35 of 35 measurements past the one-billion mark. At 150 million words it had looked free. A short probe compresses the whole learning-rate schedule into a fraction of the tokens, which is a different regime from the one a long run spends most of its time in. Short probes are good for picking a setting and unreliable for predicting what it will cost.

Final perplexity across all five: MoE 14.4025; MoE with topic fed to the router 14.4299, up 0.19%; MoE with the topic loss 14.4696, up 0.47%; dense at the same compute per word 15.5365, up 7.87%; the smaller dense reference 16.1902, up 12.41%.

I ran each configuration once. Two runs that should have been identical differed by 0.52% at one checkpoint, which is my only estimate of run-to-run noise. The 7.30% sparsity result is about fourteen times that and is safe. The 0.19% and 0.47% penalties are one to two times it, and the honest statement is that forcing topic routing produced no benefit, not that it cost a precise amount.

The one place it always failed

The version that fed topic to the router beat the control on four of five sources and still lost overall, because books sit at perplexity 23.7 against everyone else's 3 to 8 and therefore dominate the average.

Neither intervention was broadly harmful. Both failed on books, and books are the source where the label carries the least information about the content. A GitHub file is reliably code. A book contains cooking, warfare, grief, dialogue, and weather. Telling the router "this is a book" and expecting it to route accordingly forces genuinely different words down the same path.

Why the router could not be talked into it

A paper from Johns Hopkins earlier this year gives the cleanest account of what I ran into. Because MoE routers are linear maps, similarity in the hidden state is both necessary and sufficient to explain similarity in expert usage. Specialization is a property of the representation the router reads, not something the routing mechanism creates.

That predicts the strangest number I collected. Feeding the domain label straight into the router produced less topic routing, at 0.230, than never showing it a label at all, at 0.454. Handing a linear router a fact does not reorganize how it partitions; the fact just becomes another small additive term.

It also sets expectations correctly. Separate work this year found that replacing learned routing with a fixed hash costs only about 2 perplexity points against a learned router. If routing decisions barely matter compared to having the experts at all, nobody should expect better routing decisions to buy much. Which is roughly what DEMix claimed back in 2022: its reported wins were out-of-domain perplexity, modularity, and fast adaptation. In-domain quality was never the pitch.

There is a practical corollary. If you were considering a topic classifier at inference time to drive routing, the oracle label is the ceiling any such classifier could reach, and the oracle label already loses. The predictor is moot.

The thing I would check next

There is a live possibility that I measured the wrong object. My domain NMI number treats each block separately and averages the six. But a word makes six routing decisions on its way through the model, and 8 experts over 6 blocks gives 262,144 possible routes. Recent work argues that individual experts are ambiguous while whole routes are not. Knowing one digit of a six-digit code tells you very little. Knowing all six identifies it.

If that holds in my models, a per-block measurement is structurally incapable of seeing the semantic structure. The measurement is cheap, since I still have the checkpoints, and easy to get wrong: with 262,144 possible routes and roughly 524,000 words in my evaluation set, most routes appear once or never, and mutual information estimated on a table that sparse produces impressive numbers out of pure coincidence. It needs a support threshold and a shuffled-label control, or the answer is worse than no answer.

The other cheap check is depth. Convolutional networks organize from edges to textures to object parts, with early layers nearly identical no matter what you train them on. If transformers do anything similar, early blocks would route on surface form and late blocks on something more abstract, and averaging all six would report the mean of two opposite trends. I have the per-block tables already. I just threw the detail away.

---

Full write-up with all the figures: phenx.ai.