The diffusion of language models human migrations

That sentence, whose meaning is only partly true, is deliberately a garden path sentence, GPS from here on, and a particularly effective one for anyone who works with LLMs. On a first read I imagine many of you took models as the noun, as in “language models”, and only by reading on realised it was the verb: the diffusion of language shapes human migrations.

When we read a GPS our eyes make quick, imperceptible backward movements, called regressive saccades, that let us go back to what we already read and reinterpret the sentence. These sentences are rare, usually built on purpose in English, and even rarer in Italian.

That’s interesting for LLMs, because the models we use today read a sentence one token at a time, and while processing a token they can’t draw on information from the tokens that follow. This holds at every layer: the state of models will never see human migrations. The only way the model can “re-read” is for later tokens to look back at earlier ones.

But if an LLM interprets a GPS correctly, then at some point while processing the sentence it must have got the syntactic roles right. Finding out where that happens, and whether there’s a dedicated circuit, is the question I started this experiment from, using Mechanistic Interpretability tools.

Mechanistic Interpretability

It’s the field that tries to understand how LLMs behave by studying the states they go through internally while they process text. Because of their architecture and size, LLMs are among the so-called “black box” models: even knowing the whole structure, every weight and every internal state they go through, we can’t connect the operations they perform to a clear explanation.

But tools and techniques have been built that let us figure out, in specific cases, why a model makes certain decisions and predicts a particular token next.

Choosing the model

I chose the open-weights gemma-3 family, which goes from 270M to 27B parameters, and used the 1, 4, 12 and 27 billion versions. The first thing to ask is whether these models actually understand what a GPS means.

To do that I picked about 30 GPS, some written by me and some found in the literature and other online resources, all with the same noun/verb ambiguity as the title. I had each model interpret the GPS with this syntax:

Sentence: The old man the boat. In this sentence, the word “man” is a

preceded by four solved examples, and repeated with two different phrasings.

Before it completed the sentence, I extracted the logits of the words verb and noun and computed their difference, which equals ln P(verb | prompt) − ln P(noun | prompt). So it measures how much the model prefers answering “verb” over “noun”, and I use it as a proxy for the model’s understanding of the sentence. I did the same on many unambiguous sentences, to check that the measure worked.

I only accepted models that correctly solved at least 18 control sentences out of 20, both those where the word is clearly a verb and those where it’s clearly a noun. Only gemma-3:27B passed. For the others I can’t say they don’t understand GPS: it’s my measure that doesn’t work on them.

And even 27B doesn’t understand all of them: in the end only 25 sentences made it into the experiment. Curiously, it understands the title of this post, but not the most famous GPS of all, The old man the boat.

Of these 25 sentences, 12 were used to make every choice in the following experiments, which I then validated on the other 13, which I had never looked at.

Activation patching

Having chosen the difference between the verb and noun logits as a proxy for understanding, I used the same metric in all the following steps.

I started with activation patching: I had the model process sentences very similar to the GPS, but without the ambiguity, which I call nonGPS. They’re identical to the GPS up to the ambiguous word, and different from there on:

The young captain the ship. → The young captain steered the ship.

The residual stream

A clarification: while an LLM reads text, every token is assigned a vector of numbers, the residual stream, which goes through all the layers of the model (attention heads, then MLP) until the last one. There the vector of the last input token is normalised and multiplied by an unembedding matrix, and the result is the logits for every possible next token.

So if at some layer I have the vectors of all tokens, those are enough to compute everything else up to the output. Attention heads are the only structures that can move information from one vector to another, and always only from an earlier token to a later one.

Activation patching means implanting a vector, or several, into the residual stream, to see how this changes the final result compared to the original state: it replaces one of these vectors at some layer and lets the following layers recompute.

Specifically, I implanted activations from the nonGPS on several important words: the ambiguous ones like captain, the disambiguating ones like the/steered, the end of the sentence, the ambiguous word quoted in the question and the last word at the end of the question. One position and one layer at a time, in both directions, watching how each patch affected the metric.

The most important position to change is the disambiguating one (the/steered), the only one inside the GPS with a real impact on the final result, partly because the words before it don’t have information about the rest of the sentence yet: on captain the effect is exactly zero. You can also see it slowly lose importance between layers 22 and 28, and from 29 onwards replacing it has practically no impact.

The importance doesn’t disappear: the information is simply moved further along the text. Around layer 29 it reaches the word I ask about in the question, and in the last layers it’s entirely concentrated in the last token received, the one right before the choice between “noun” and “verb”, which goes through the unembedding matrix to produce the logits.

This is a first sign that there’s a path mediated by attention heads, the only structures that can move information between two different vectors in the residual stream, carrying the information from the disambiguating word to the output.

Attention heads

Looking for confirmation, I applied the patch not to the whole residual stream but only to the outputs of single attention heads, at the word quoted in the question. I expected almost all the effect to be concentrated in the heads of the layers highlighted before.

And indeed, transplanting the output of all 416 heads of layers 20–32 brings back 58% of the difference, and 4 heads alone (L23H18, L29H25, L29H18, L29H30) bring back about 70% of that, while 4 random heads do almost nothing, about 1%. L23H18, from the word in the question, even puts 90% of its attention on the disambiguating word.

The four heads were chosen on the 12 development sentences only, with a rule fixed before seeing the results. On the 13 sentences I had kept aside they bring back 41% of the verb reading, in all 13 sentences, while random heads or a random transplant of the same strength stay around 1%.

Ablation

Next I tried another technique: ablation. In a rather blunt way, it means removing a piece of the model and seeing how this changes the final result. Since activation patching had given me a list of attention heads that carry most of the information from the disambiguating word, I tried with those.

Switching off the four heads together does affect the metric. But testing it on unambiguous sentences too shows a similar, smaller impact, and the noun sentence moves towards zero as well: every answer becomes less sure. So this may be a generic circuit, carrying the syntactic role of a word where it’s needed, and not something specific to GPS.

What I found, and what I didn’t

To sum up: inside gemma-3:27B the information that tells “verb” from “noun” is born on the disambiguating word, stays there until about layer 28, and is then copied by some attention heads, four of which do a good part of the work, first onto the word I ask about in the question and then onto the last token, the one the model answers from. This was confirmed on sentences I had never used.

What I didn’t find is a reinterpretation circuit: the heads that carry the information work the same way on unambiguous sentences, and no part of the model seems to serve GPS alone. It seems the model doesn’t need a special mechanism: the “second thought” is already all in the state of the disambiguating word.