Jul 16, 2026

After The Bubble - Snake I

I’ve expanded the GGUFun project: I started by generating a deterministic GGUF file that always produced the same exact sentence, then made it the parodistic website “After the bubble”, a catalog of GGUF files for after the AI bubble bursts, for all the ollama servers around the world that will be left unused.

While the GGUFs I hand crafted with the help of coding agents are preposterous, I did learn a thing or two while doing it, while it’s important to understand that no GGUF here contains any value that has been “learned”, only manually set.

Snake

I’ll start with the first game GGUF I built: Snake. But I got there by building something else first.

Point on a grid

Removing all additional features, the basic necessity for a working Snake game is that directional commands allow the coordinates of a point in a grid to move, and that’s what I started implementing: a single point on a 4x4 grid.

Communication

The frontend can send a message to the ollama server, either “<M:L>”, “<M:R>”, “<M:U>”, or “<M:D>”, the directions the point can move.

As a response, the server will answer with “<P:{pos}>”, where pos represents the current updated position of the point in the grid (y * 4 + x).

Since the model is stateless, and thus has no memory, it needs to read the current state from the conversation: each time, it reads the last token, representing the move, and the second to last token, representing the state before the move.

Residual stream

Each residual stream is structured so that it has the following non communicating blocks, all one hot encoding of some value:

CELL: the current cell (16 values for the 4x4 grid)
MOVEMENT: the movement (4 values)
G1: the previous cell, computed from attention (16 values for the 4x4 grid)
NH: the next cell, computed from MLP (16 values for the 4x4 grid)
POSB: absolute position from the position embeddings (64 values, max context size)

The total used size for the embeddings is 116, so a size of 128 is chosen.

Attention to move information forward

Here comes the reason why gpt2 was the chosen architecture for this and the following GGUFs: position embeddings are added directly to the residual stream, and here are set so that they are one hot encoding of the position in the conversation.

Then there is a single attention head, that takes the following values for each token at position k: one_hot(k - 1) for the query, and one_hot(k) for key. The query is scaled by a large factor, ~60·√head_dim, compensating llama.cpp’s 1/√head_dim scaling, so the softmax result are actually close to an actual one hot encoding.

The dot product of those two values is therefore ~1 only for the token pair (k, k - 1), so each token attends only to the direct predecessor, copying the CELL info from the predecessor into its G1 block.

MLP as AND gates

Each MLP neuron acts for a single pair cell/move, we need then 16 * 4 = 64 neurons: each neuron will fire only for the correct pair.

What makes a neuron an AND gate is the bias: each neuron receives a weight S from its cell dimension (in G1) and from its move dimension (in MOVEMENT), plus a bias of -S * threshold, so that the sum crosses zero only when both dimensions are active and the GELU lets the value through.

When a neuron fires it writes a one hot in the NH block, at the position of the next cell for its (cell, move) pair: the choice of which dimension each neuron writes to works as transition table for this state machine.

Unembedding and Layer Norm

The final part: the unembedding reads the NH block and uses it to produce the next state: the position of the point in the grid after the movement.

The threshold for the neurons cannot be computed on paper: LayerNorm normalizes each token over the whole residual stream, so the actual post-norm magnitude of the G1 and MOVEMENT values depends on everything else sitting in the row.

So it is measured instead, with a generator playing a bunch of random walks, recording the inputs of the neurons that must fire and of those that must not, and placing the threshold in the middle of the gap between the two groups.

A nice schema

flowchart LR
    WTE["token embedding"]
    WPE["position embedding"]

    subgraph RS0["residual stream · input"]
        direction TB
        CELL0["CELL · 16
current cell"]
        MOV0["MOVEMENT · 4
move"]
        G10["G1 · 16
empty"]:::empty
        NH0["NH · 16
empty"]:::empty
        POSB0["POSB · 64
absolute position"]
        CELL0 ~~~ MOV0 ~~~ G10 ~~~ NH0 ~~~ POSB0
    end

    ATT["attention
offset -1 selector"]

    subgraph RS1["residual stream · after attention"]
        direction TB
        CELL1["CELL · 16
current cell"]
        MOV1["MOVEMENT · 4
move"]
        G11["G1 · 16
previous cell"]
        NH1["NH · 16
empty"]:::empty
        POSB1["POSB · 64
absolute position"]
        CELL1 ~~~ MOV1 ~~~ G11 ~~~ NH1 ~~~ POSB1
    end

    MLP["MLP
64 AND gates"]

    subgraph RS2["residual stream · after MLP"]
        direction TB
        CELL2["CELL · 16
current cell"]
        MOV2["MOVEMENT · 4
move"]
        G12["G1 · 16
previous cell"]
        NH2["NH · 16
next cell"]
        POSB2["POSB · 64
absolute position"]
        CELL2 ~~~ MOV2 ~~~ G12 ~~~ NH2 ~~~ POSB2
    end

    OUT["unembedding"]
    GEN["logit of <P:new>"]

    WTE -- "<P:c>" --> CELL0
    WTE -- "<M:m>" --> MOV0
    WPE --> POSB0

    POSB0 -. "query + key" .-> ATT
    CELL0 -. "value (from the predecessor)" .-> ATT
    ATT -- "writes" --> G11
    RS0 ==>|"residual: other blocks copied"| RS1

    G11 -. "cell" .-> MLP
    MOV1 -. "move" .-> MLP
    MLP -- "writes" --> NH2
    RS1 ==>|"residual: other blocks copied"| RS2

    NH2 -.-> OUT
    OUT --> GEN

    classDef empty stroke-dasharray: 5 5,opacity:0.6;

Conclusion

I’ll explain snake in a later post, this one is quite long as it is.