What does a Mixture-of-Experts router actually read?

A few weeks ago I built a small tool called J-Scope. It reads the concepts a language model is holding mid-computation: not the words it is about to say, but the ideas sitting in its residual stream a dozen layers deep. Give a 4B model the riddle "the currency of the country shaped like a boot," and J-Scope shows you the model thinking Italy on the word "boot," well before it writes euro. The word "Italy" is nowhere in the prompt. The model put it there.

That tool answered a question about the model's output. It left me with a different one, about a part of the model I had been ignoring. A Mixture-of-Experts model does not run all of its weights on every token. Each layer holds a bank of experts, 64 of them in the model I use here, and a small router picks a handful, eight, to actually run. The other 56 sit out. The router makes that choice by looking at the token's residual stream, the exact same vector J-Scope reads.

So the router and my lens are reading the same thing. Which means I can point the lens at the router's decision and ask a question people usually answer with a shrug. What is the router reading? What makes it send a token to one expert and not another?

The common answer is concept. You have met the folk model even if nobody stated it to you: there is a code expert, a math expert, a French expert. The router reads what the token is about and hands it to the specialist. It is a clean story, and MoE diagrams reinforce it. I spent a week checking it on two models. The story is wrong. The router is not reading meaning. It is reading form.

https://video.wixstatic.com/cf00a8_7ff5c6aed81345a19998dbce62259568

The whole investigation in 90 seconds, narrated. Every chart below animates from the measured result.

Read the router directly

J-Scope reads the concept a token is carrying; the router reads that same token to choose its experts. So I read the concept flowing into each expert, across thousands of tokens, and built up, for every expert, the distribution of concepts it receives. A concept specialist would show a peaked, coherent distribution: a geography expert fed Italy, Tokyo, river, border. A non-specialist would be fed whatever comes. To separate real specialization from the concentration you get by luck, I measured each expert against a random sample of the same size. The gap between them is the real signal.

The experts are selective. The gap is positive at every layer and grows with depth. But look at what the selective ones specialize in. One expert almost only sees closing quotes and brackets. Another concentrates on newlines. Another on symbols. Another on subword fragments. These are not concepts. They are shapes. The router built a bracket expert, a newline expert, a symbol expert. Nothing that reads like "geography" or "biology," nothing about meaning at all.

The most selective experts are all shape and grammar. None sorts a topic.

Maybe OLMoE is just unusual

A result on one model is a fact about that model until you check another. So I fit a second lens, for Qwen1.5-MoE, a different lab and a deliberately different design: sixty small experts plus an always-on shared expert, an architecture built specifically to encourage specialization. If any MoE were going to route by concept, this is the one that should.

It came out the same. Same shape to the numbers, same climb with depth, same kind of experts. And Qwen handed me the cleanest example in the study: an expert whose job, across the whole corpus, is the words would, is, were, are, was. That is not a topic. It is a grammar, the auxiliary and copula verbs, sorted into one bin. The design meant to push experts toward meaning pushed them toward parts of speech.

Experts get more selective with depth, and two labs' models agree on the shape and the ceiling.

The last place meaning could be hiding

There is a fair objection. The lens's top read for an ordinary token is usually that token's own surface form, so a method that reads token forms will, unsurprisingly, find experts sorting token forms. Maybe the semantic sorting is real and buried under all the brackets and suffixes. So I threw out the brackets and suffixes. I kept only tokens where the lens reads a real content word, and asked the same question of just those.

Content words are still sorted more than chance, so the router does not ignore them, but what it sorts them by is once again grammar: on Qwen the strongest content experts collect participated, died, ended, moved, sailed, arrived, past-tense verbs, not a subject. There was one honest exception. On OLMoE a cluster of experts lit up for dorsal, fins, spines, species, genus, real marine-biology vocabulary, words that share meaning and not spelling. For a moment that looked like a genuine topic expert. Then I checked Qwen, and it was not there. My corpus had a fish article in it, and an encyclopedia article drops all its fish words in one place, so they route together, not because an expert knows fish but because they showed up together. Controlling for word frequency does not control for topic. Checking a second model does, and it said mirage.

Strip out symbols and subwords and what's left still sorts by grammar, not topic.

What a Mixture-of-Experts router is actually doing

Two models, two labs, one of them built to specialize, all saying the same thing. Routing sorts token form, first the lexical layer of brackets and symbols, then syntax and morphology, function words and verb tense. It does not sort meaning. The tidy picture of a geography expert beside a code expert is a diagram, not a mechanism.

That is less deflating than it sounds. It does not say the experts are useless or interchangeable; they plainly do specialized work. It moves where the specialization lives. The model is not deciding "this is a French token, send it to the French expert." It sorts by surface structure and does the semantic work in the parts of the computation that every token shares. The meaning was never in the choice of expert; it lives in the computation they all share.

This is not the first sighting of syntactic routing, and I want to be clear about that. When Mistral analyzed Mixtral's router, they were blunt about it: "we do not observe obvious patterns in the assignment of experts based on the topic." Papers on arXiv, biology abstracts, and philosophy all routed almost identically, at every layer. What they found instead was structural, the word self in Python and the indentation tokens in code landing on the same expert, consecutive tokens sticking to one expert far more often than chance. Earlier, ST-MoE catalogued its experts by hand and got a list that reads like a grammar textbook: a punctuation expert, a conjunctions-and-articles expert, a verb expert, a numbers expert. The texture is known.

Two things here are mine to add. The method: instead of eyeballing which tokens land on which expert, I read the concept in the residual stream with a Jacobian lens and scored every expert against a random-token baseline, so "selective" means selective beyond chance, and it is a number I can plot. And a sharpening: ST-MoE found that its decoder experts barely specialized at all, but that was an encoder-decoder model read by eye. On two decoder-only MoEs, measured against that baseline, the selectivity is plainly there. It just sorts form, not meaning. Add the small discipline that saved me from the fish: check a suspicious specialist against a second model before you believe it.

---

The lens is Anthropic's Jacobian lens, open source, and it fits onto any Hugging Face MoE with a one-line adapter. Fit one for your own model, aim it at its router, and see what it is reading. My bet is that it is not what you would draw on the whiteboard.