I gave a real language model the most quiz-show sentence I could think of and asked it to finish:

prompt: 'The capital of France is'

  8.5%  ' the'
  4.8%  ' now'
  4.6%  ' a'
  3.2%  ' France'
  3.2%  ' Paris'

Paris is fifth. GPT-2 assigns 8.5 percent to ' the' and 3.2 percent to ' Paris'. The output is not one selected answer. It is a distribution across 50,257 possible next tokens, and the table shows its five largest entries.

I reran that output before trusting the preserved transcript. The current run uses the Hugging Face gpt2 snapshot at revision 607a30d783dfa663caf39e06633721c8d4cfcd7e; its model.safetensors SHA-256 is 248dfc3911869ec493c76e65bf2fcf7f615828b0254c12b473182f0f81d3a707. The interpreter is Python 3.10.14 with Torch 2.9.1, Transformers 4.57.1, and NumPy 1.26.4. It ran on the M4 CPU with four Torch threads. MPS was unavailable in this environment. The top five reproduced exactly.

Those versions matter. Transformers 4.57.1 selects an optimized attention path by default that does not return attention weights, so the attention probes explicitly select its eager implementation. The model has 124,439,808 unique parameters. Its input embedding and output projection share the same storage. Both details come from the loaded object rather than the model name.

The original investigation made two claims that the code did not earn. It treated a zero-based intermediate rank of 1 as though Paris were the top prediction. France was top; Paris was second. It also said the model had been stored and executed as packed 4-bit integers. The quantization probe rounded weights to 4-bit codes and immediately reconstructed them into fp32 tensors. It measured approximation error, not packed storage, a dequantization kernel, or a speedup. Keeping those distinctions visible changes the route through the same forward pass.

Text is not what the model sees

The model receives integers rather than Unicode text. GPT-2’s tokenizer first maps UTF-8 bytes through a reversible byte-to-Unicode table, then applies learned byte-pair merges. OpenAI’s archived encoder.py contains that path. The resulting vocabulary has 50,257 entries, and this input becomes five ids:

prompt:     'The capital of France is'
token ids:  [464, 3139, 286, 4881, 318]
tokens:     ['The', ' capital', ' of', ' France', ' is']

Five tokens. Look closely, because two details here trip up everyone. First, the tokens aren’t words split on spaces in the way you’d guess, the leading space is part of the token. ' France' (id 4881, with its space) is a different token from 'France' at the start of a sentence, and the model learned both. Second, common words are single tokens but rare ones shatter into pieces; “France” survived whole because it’s frequent, but a word like “Ciechanowski” would come apart into four or five fragments. This is why token counts never match word counts, why models have a “context length” measured in tokens not words, and why every API bills you per token. The text became five integers, and from here on the model only knows the integers.

I made those two claims (leading space matters, rare words shatter) the way you’d expect a blog post to make them, confidently and without evidence, so let me actually run the tokenizer and see if I was right, because “would come apart into four or five fragments” is exactly the kind of sentence people invent:

'France'        -> [28572]        ['France']
' France'       -> [4881]         [' France']
'Ciechanowski'  -> 4 tokens       ['C', 'ie', 'chan', 'owski']
'antidisestablishmentarianism' -> 5 tokens ['ant','idis','establishment','arian','ism']

Both claims survive the rerun. 'France' with no leading space is token 28572, while ' France' is 4881, so the embedding lookup selects a different row. “Ciechanowski” becomes exactly four pieces, ['C', 'ie', 'chan', 'owski']. Those boundaries are neither syllables nor a grammatical analysis. They are consequences of the tokenizer’s byte representation and merge table.

The word does not jump from twelve characters to four pieces in one lookup. I replayed the merge loop. UTF-8 encoding first gives twelve bytes for this ASCII word. GPT-2’s reversible byte table represents every possible byte as a Unicode character that can safely pass through the rest of the tokenizer. In this case the visible characters happen to remain unchanged:

initial symbols:
['C','i','e','c','h','a','n','o','w','s','k','i']

The vocabulary file includes an ordered list of merge pairs. A lower rank means a pair was learned earlier and takes priority. On each iteration, the algorithm finds adjacent pairs currently present in the symbol list, selects the available pair with the lowest rank, and merges every matching occurrence. The actual trace is:

step 1  rank    16: 'a'   + 'n'  -> ['C','i','e','c','h','an','o','w','s','k','i']
step 2  rank    66: 'o'   + 'w'  -> ['C','i','e','c','h','an','ow','s','k','i']
step 3  rank    98: 'c'   + 'h'  -> ['C','i','e','ch','an','ow','s','k','i']
step 4  rank   238: 'i'   + 'e'  -> ['C','ie','ch','an','ow','s','k','i']
step 5  rank  1410: 'ow'  + 's'  -> ['C','ie','ch','an','ows','k','i']
step 6  rank  2891: 'ch'  + 'an' -> ['C','ie','chan','ows','k','i']
step 7  rank  3850: 'k'   + 'i'  -> ['C','ie','chan','ows','ki']
step 8  rank 11823: 'ows' + 'ki' -> ['C','ie','chan','owski']

No ranked merge remains between the four final symbols, so they map to ids [34, 494, 3147, 12079]. A tempting implementation error is to scan the merge list once and join any pair that appears. That fails because each merge creates new adjacent pairs. After o and w become ow, the pair ow plus s becomes available. After k plus i becomes ki, the pair ows plus ki becomes available. The candidate set must be recomputed after a merge.

Another error is to begin from Unicode code points rather than bytes. UTF-8 can represent one visible character with several bytes. GPT-2’s byte mapping means any input has a representation even if a whole character or word never appeared in the learned vocabulary. The merges can later join byte representations into larger units. This avoids an unknown-token hole, but it means a rare character can consume several token positions.

The leading space also enters before the merge loop. The tokenizer’s regular expression groups a space with many following word pieces, then the byte representation preserves that space through merging. That is how ' France' and 'France' acquire different ids without a special word-boundary flag. The boundary is in the token string itself.

This mechanism explains the observed pieces without assigning them linguistic meaning. chan exists as a token because its byte sequence participated in useful merges across the training text. It need not be a morpheme in this surname. Once tokenization finishes, the transformer receives only the four ids. It does not retain a side table saying which merges produced them.

The place this gets genuinely weird is numbers, and it matters more than it looks:

'1998'       -> 1 token    ['1998']
'127.0.0.1'  -> 7 tokens   ['127', '.', '0', '.', '0', '.', '1']
' 3.14159'   -> 4 tokens   [' 3', '.', '14', '159']

1998 is one token in this vocabulary, while 3.14159 is four. The model must learn numeric relationships through these token sequences rather than receiving a built-in floating point quantity. That representation can make arithmetic harder, but this probe does not isolate it as the cause of GPT-2’s arithmetic failures. Model capacity, training data, objective, and decoding all remain confounded. The only measured word-to-token ratio here is 23 tokens for one 20-word sentence, or 1.15. A general billing estimate needs a representative sample from the actual language and content being served.

the expected answer was one token to the right

Nothing in the input marks it as a question. There is no field saying country=France, no slot named capital, and no requested answer type. The five ids are a prefix, exactly like the beginning of any other piece of text. GPT-2 was trained to assign probabilities to the token that follows a prefix. The original GPT-2 technical report describes that language-model objective and the WebText training corpus.

For a tiny sequence of four ids

[41, 82, 17, 63]

training can form three input and target pairs without any human labels:

input prefix       target token
[41]                    82
[41, 82]                17
[41, 82, 17]            63

Implementations process these positions together. They feed [41, 82, 17], produce a probability row at each position, and compare those rows with [82, 17, 63]. This is often called shifting the targets by one position. The causal mask is what stops the first output row from reading ids 82 or 17 while it predicts 82.

Suppose the probability assigned to the actual next token is p. The loss for that position is the negative natural logarithm:

loss = -log(p)

The logarithm turns multiplication of probabilities across a sequence into addition. If two actual tokens receive probabilities p1 and p2, their joint probability under the model is p1 × p2, while their negative log likelihood is

-log(p1 × p2) = -log(p1) + -log(p2)

This gives each position an additive contribution. A probability near one contributes a value near zero. A small probability contributes a larger penalty. It does not matter whether the token is a fact, punctuation, part of a name, or an uninteresting space-prefixed word. The objective sees an integer that occurred next.

For the opening distribution, the exact probabilities and losses are:

candidate target    probability    negative log likelihood
' the'               0.084591              2.469926
' France'            0.032377              3.430321
' Paris'             0.032243              3.434449

If the training text at that position continued with Paris, the third number would contribute to the loss. If it continued with the, the first would. A training step does not declare the other 50,256 tokens impossible. It adjusts parameters so the observed token receives more probability relative to the alternatives, averaged with many other positions.

The loss over a batch is usually the mean of these position losses. Calling that cross entropy introduces one more term but no new operation for a one-hot target. A one-hot target is an array with one 1 at the observed token id and zeros elsewhere. Multiplying it by the negative log probabilities and summing selects the one entry shown above:

target = [0, 0, 1, 0]
negative log probabilities = [1.2, 0.4, 2.7, 3.1]

cross entropy = 0*1.2 + 0*0.4 + 1*2.7 + 0*3.1 = 2.7

Backpropagation, which belongs to the backward investigation, determines how each parameter affected that loss. The forward path here stops earlier. It computes the logits, converts them to probabilities for inspection, and returns the row. A generation policy then decides what to do with it.

Greedy generation chooses the largest logit, so this row produces the. Sampling treats the probabilities as draw weights, so Paris remains possible. Temperature divides logits by a positive value before softmax. A value below one magnifies score differences and concentrates probability; a value above one flattens them. Top-k and top-p policies discard part of the distribution before sampling. Those policies can change the emitted token without rerunning the transformer blocks.

This separates three facts that are easy to collapse into one. Training rewarded next-token probabilities on observed text. The forward pass returned a distribution for this prefix. Decoding may select one entry from that distribution. “The capital of France is” resembles a question to a person, but the model interface used here never received an instruction to answer it. Paris being fifth is therefore not a contradiction inside the objective. It is an observation about one row produced by that objective.

fifth place hid fifty thousand alternatives

A rank gives order but discards distance. Paris is fifth, but that statement alone cannot distinguish 3.2% from 0.000032%. The top five probabilities in the opening sum to 0.243321. The other 50,252 tokens collectively hold 0.756679. Most probability mass is outside the displayed table.

Softmax makes the distance between two logits interpretable without considering the other 50,255 logits. For tokens a and b,

p(a) / p(b)
= [exp(logit(a)) / total] / [exp(logit(b)) / total]
= exp(logit(a) - logit(b))

The common denominator cancels. The measured logit for the exceeds the logit for Paris by 0.964523. Exponentiating that gap gives 2.623537, the same as dividing their exact probabilities:

0.084591 / 0.032243 = 2.623537

Adding 100 to every logit would leave that ratio unchanged. Multiplying every logit by two would square every pairwise odds ratio and make the distribution more concentrated. This is why temperature divides logits rather than probabilities. A temperature of two halves the logit gap and changes this pair’s odds from exp(0.964523) to exp(0.482262), about 1.62. Renormalization then changes every probability together.

Entropy describes the spread of the complete row:

entropy = -sum(p(token) × log2(p(token)))

A distribution with two tokens at 0.5 each has one bit of entropy:

-(0.5 × -1 + 0.5 × -1) = 1 bit

A uniform distribution over all 50,257 tokens would have log2(50,257) = 15.617 bits. The opening row measures 8.654 bits. Raising two to that value gives 402.84, so it has the same entropy as a uniform distribution over roughly 403 choices. That does not mean exactly 403 tokens are possible. A long tail of very small probabilities can have the same entropy as fewer equal probabilities.

Entropy, top-five mass, and rank answer different questions. Rank says four tokens have a higher score than Paris. Top-five mass says the displayed rows contain only 24.3% of the draw probability. Entropy says the whole distribution is far from both a single certain choice and a uniform vocabulary. None says whether Paris is factually correct. Correctness requires an external target and an evaluation rule.

This matters when comparing interventions later. KL divergence can report that two full distributions differ even if their top token is the same. Top-one agreement can report failure when two nearly tied logits exchange order, even if the distributions are almost identical. Perplexity can worsen while a particular capital answer improves. A measurement has to match the decision being made.

the first grid was only repeated multiplication

The next line of model code uses tensors. That name can make ordinary arithmetic look like a separate subject. For the path here, a tensor is an array with a recorded shape. A vector is a one-dimensional array. A matrix is a two-dimensional array. GPT-2 adds batch, token, and head dimensions later, but every multiplication still reduces to products and sums.

Start with a dot product:

[1, 2, 3] dot [1, 0, 1]
= 1*1 + 2*0 + 3*1
= 4

The two vectors need the same length because every coordinate on the left must pair with one coordinate on the right. The result is one number. A matrix multiplication repeats that calculation for every row of the left matrix and every column of the right matrix:

[[1, 2, 3],        [[1, 0],        [[ 4,  5],
 [4, 5, 6]]    x    [0, 1],    =    [10, 11]]
                       [1, 1]]

The top left result is the first row dotted with the first column: 1*1 + 2*0 + 3*1 = 4. The top right is the first row dotted with the second column: 1*0 + 2*1 + 3*1 = 5. The second output row repeats those two dot products with [4, 5, 6].

The shapes describe which dimensions survive:

(2 rows, 3 columns) x (3 rows, 2 columns) = (2 rows, 2 columns)
          ^                  ^
          these inner dimensions must match

If the input contains T token rows and each row has width D, its shape is (T, D). Multiplying by a weight matrix shaped (D, H) produces (T, H). Every token gets an output row. Every output coordinate combines all D input coordinates.

Here is the complete list-based implementation behind the printed matrix:

def dot(left, right):
    return sum(a * b for a, b in zip(left, right))

def transpose(matrix):
    return [list(column) for column in zip(*matrix)]

def matmul(left, right):
    columns = transpose(right)
    return [[dot(row, column) for column in columns] for row in left]

This is not pseudocode. The probe runs it and asserts [[4, 5], [10, 11]]. Torch performs the same mathematical operation through optimized native kernels with different blocking, vectorization, threading, and floating point grouping.

A matrix multiplication shaped (M, K) x (K, N) computes M*N output numbers. Each output uses K multiplications and roughly K additions. Counting one multiplication and one addition as two floating point operations gives approximately

2 * M * K * N operations

The units are operations, not bytes or seconds. Turning that into time requires a measured execution rate and enough work to keep the arithmetic units occupied. The same operation can instead wait on memory when its matrices are too small, poorly reused, or fetched repeatedly.

An embedding lookup is different. The token id 4881 does not multiply by every row in a 50,257 by 768 table. It selects row 4881. Five input ids therefore produce a matrix with shape (5, 768). The row selection moves 5 * 768 fp32 values in this CPU run, which is 15,360 bytes. The entire table remains available for other ids and later reappears as the output projection.

At the other end of the model, the final 768 values are multiplied by the transposed embedding table. The shape is

(1, 768) x (768, 50,257) = (1, 50,257)

The 50,257 outputs are called logits. A logit is an unnormalized score. It can be negative, positive, or shifted by the same constant as every other logit without changing the probabilities. Softmax performs that conversion:

def stable_softmax(values):
    maximum = max(values)
    shifted = [exp(value - maximum) for value in values]
    total = sum(shifted)
    return [value / total for value in shifted]

Exponentiation makes score differences into positive ratios. Division by their sum makes the results add to one. Subtracting the maximum changes neither ratio because every numerator and the denominator receive the same common factor. It prevents an avoidable overflow:

naive exp([1000,1001,1002]) -> overflow
subtract max first          -> [0.090031, 0.244728, 0.665241]
sum                         -> 1.000000

That negative control is useful because “numerically stable” names a concrete property here. The naive formula cannot represent exp(1000) as a finite binary64 value. The shifted formula only evaluates exp(-2), exp(-1), and exp(0), yet returns the same mathematical distribution.

The opening percentages are one such 50,257-way softmax. Selecting the largest probability would produce a greedy next token. Sampling would draw according to the whole distribution. Temperature would rescale the logits before softmax and change how concentrated the distribution is. None of those decoding choices changes the forward pass that produced the logits.

An integer becomes a direction in space

An id like 4881 is meaningless as a number, it’s just a row index. The first thing the model does is look that row up in a giant table called the embedding matrix, turning each token id into a vector: a list of numbers that is the token’s actual representation. In GPT-2 that table is one of the model’s biggest single pieces:

wte shape: (50257, 768)  -> each token is a 768-dim vector
token 'The' -> vector[:6] = [-0.069, -0.02, 0.064, -0.062, -0.114, -0.062]

Fifty thousand rows, one per vocabulary entry; 768 columns. Token “The” selects one point in a 768-dimensional space. The values were learned during next-token training rather than assigned by a human. Nearby rows can reflect shared usage, spelling, syntax, or other correlations. Calling every geometric relation “meaning” would assign an interpretation the distance calculation does not provide.

GPT-2 adds a learned position row with the same width to each selected token row. Position 0 adds one vector, position 1 another. Addition preserves the (5, 768) shape while making identical tokens at different positions distinguishable. The input to the first block is therefore

token embedding[token_id] + position embedding[position]

I have never trusted the king - man + woman ≈ queen claim, because I’ve watched it get repeated for a decade mostly by people quoting word2vec, which is a different model trained a different way, and GPT-2’s embedding table was trained to predict next tokens, not to be a nice analogy space. So I ran it, right here, in this model’s actual wte. I took the row for king, subtracted man, added woman, and asked which vocabulary tokens sit nearest the resulting point by cosine similarity:

king - man + woman, nearest tokens (excluding king/man/woman):
   0.709  ' queen'
   0.605  ' princess'
   0.596  ' Queen'
   0.593  ' kings'
cosine(result, queen) = 0.709;  queen ranks #1 of 50257

The printed neighbor function excludes the three input tokens. Under that condition, queen is the nearest remaining row. Its zero-based rank among the complete 50,257 rows is 1, meaning one row still ranks above it. The original prose called rank 1 first place and silently changed the candidate set. The qualified result is still interesting: after excluding the inputs, queen leads at cosine 0.709. Before the vector arithmetic, queen was already near king at cosine 0.657. The subtraction and addition improve an existing neighborhood relation on this one analogy; they do not establish a general algebra of concepts.

The geometry that did surprise me was the capital. I asked for the nearest neighbors of Paris:

' Paris' nearest:  'Paris' 0.834    ' France' 0.634   ' French' 0.581
                   ' London' 0.579  ' Copenhagen' 0.570

The no-leading-space token 'Paris' is the closest different vocabulary row, a fact the earlier table omitted. ' France', ' French', ' London', and ' Copenhagen' follow. The cosine between ' Paris' and ' France' is 0.634, while the mean against one thousand seeded random token ids is 0.242. The rows contain a measurable relation before the transformer blocks run. That does not prove the embedding alone stores the fact “Paris is the capital of France.” It establishes a correlation the later computation can use.

One more thing about that second vector, the position embedding, because I measured something I didn’t expect and it turns out to matter three sections from now. Position embeddings are supposed to be small nudges that tell the model where each token sits. I checked their magnitudes:

mean ||token emb|| = 3.96,  mean ||pos emb|| = 3.39
pos emb norm at positions 0..5: [9.88, 5.19, 4.59, 4.34, 4.18, 4.06]

Positions 2 through 5 have norms around 4. Position 0 has norm 9.88. That is a direct measurement of the learned table. Norm alone does not reveal why training produced it or whether it causes later attention to favor position 0. The attention intervention below is enough to reject my earlier claim that the first position was merely a harmless drain.

Attention: how a word looks at the words around it

Now the interesting part, the one mechanism that made transformers work. Each token’s vector, so far, only knows about that token. “is” doesn’t yet know it’s the “is” that follows “France.” Attention is how each token reaches out to the other tokens and pulls in the context it needs.

Each token row is multiplied by three learned matrices. The results are called query, key, and value. Those names are roles in an equation, not database fields:

Q = X Wq
K = X Wk
V = X Wv

attention(X) = softmax((Q K^T) / sqrt(head_width) + mask) V

Q K^T compares every query row with every key row. If there are T tokens, the score matrix is (T, T). The mask sets a future position’s score to negative infinity before softmax, making its probability zero. Multiplication by V turns each probability row into a weighted sum of value rows. This is the scaled dot-product mechanism in Attention Is All You Need.

The list-based probe uses three token rows with width two. Its learned matrices are deliberately tiny:

X = [[1,0], [0,1], [1,1]]
Q = [[1,0], [1,1], [2,1]]
K = [[1,0], [0,1], [1,1]]
V = [[1,2], [2,-1], [3,1]]

The unscaled score matrix is

Q K^T = [[1,0,1],
         [1,1,2],
         [2,1,3]]

Row 0 is not allowed to use key 1 or 2, so its causal softmax is [1,0,0]. Row 1 can use keys 0 and 1. Their scaled scores are equal, producing [0.5,0.5,0]. Row 2 can use all three:

row 2 weights = [0.283995, 0.140029, 0.575975]
weighted V    = [2.291980, 1.003937]

Every row sums to one, and every weight above the causal diagonal is exactly zero. There is no special attention primitive hiding beneath the notation. The probe performs matrix multiplication, stable softmax, and another matrix multiplication.

GPT-2 applies that arithmetic at width 64 inside each head. Here are fresh attention weights for the token “is” in layer 0, head 0:

attention (layer 0, head 0) -- token 'is' attends:
  61.0%  -> 'The'
  15.2%  -> ' capital'
   6.3%  -> ' of'
   9.4%  -> ' France'
   8.1%  -> ' is'

Sixty-one percent of “is“‘s attention, in this particular head, went to “The.” The weights sum to 1, it’s a softmax, a blend, and notice it only attends to tokens at or before its own position, never after. That’s the causal mask: a token predicting what comes next is not allowed to peek at the future, so the future tokens’ weights are forced to zero before the softmax. This is what makes the model able to generate left-to-right; every position is simultaneously being trained to predict its successor using only what came before it.

The scaling is 1/sqrt(64) = 0.125. If independent query and key components each have mean zero and variance one, one product has variance one and a sum of 64 such products has variance 64. Its standard deviation grows as sqrt(64). Dividing by that value keeps the score scale roughly constant as head width changes. The Transformer paper motivates the factor because large dot products push softmax into regions with very small gradients. This variance argument uses idealized independent components; trained query and key values need not satisfy it exactly. The numerical overflow fix remains subtracting the row maximum before exponentiation, as in the earlier softmax probe. The repaired link to the floating point investigation shows why those are different concerns.

Twelve heads, looking at twelve different things

That was one head. GPT-2’s attention runs twelve of them in parallel per layer, each with its own smaller 64-dimensional query/key/value space (12 heads × 64 = 768, the full width split up), and the reason is that different heads learn to look for different relationships. Head 0 above was fixated on the sentence’s first word. But I asked which of the twelve heads paid the most attention to “France” when processing “is,” and got a completely different head:

of 12 heads, head 6 attends most to ' France' from 'is': 35.3%

Head 6 gives a third of its routing weight to “France.” That does not prove the head represents France, the subject, or any named linguistic relationship. Its output is a weighted blend of value vectors, then another learned projection mixes all twelve heads before the result rejoins the running state. The table tells me where the coefficients are large on one prompt. It does not tell me what information lives in those value vectors or how the output projection uses it. Running twelve such calculations in parallel is multi-head attention. GPT-2 splits its width of 768 into twelve spaces of width 64, concatenates their twelve outputs, and projects the combined 768 numbers back to width 768.

I’ve quoted two of the twelve heads now, head 0 and head 6, which is the coward’s way to talk about multi-head attention, so let me print all twelve. For each head in layer 0, here’s where the token “is” sends the most attention, how concentrated that head is (its entropy in bits, low means peaked, high means spread out), and how much of its attention lands on the first token:

head  argmax target   weight   entropy(bits)  % to pos0 'The'
   0        'The'      61.0%        1.71          61.0%
   1        ' is'      97.8%        0.18           0.3%
   2        'The'      46.9%        1.97          46.9%
   3        ' is'      79.8%        1.04          10.6%
   4        ' is'      27.4%        2.25          27.2%
   5        ' is'      91.9%        0.49           6.2%
   6     ' France'     35.3%        1.92          21.7%
   7        ' is'      33.3%        2.22          23.7%
   8        ' is'      47.3%        1.84          23.7%
   9        'The'      42.3%        2.09          42.3%
  10        'The'      40.1%        2.00          40.1%
  11        'The'      46.9%        2.04          46.9%

This is stranger than “twelve heads, twelve relationships.” Head 1 puts 97.8% of its coefficient mass on “is” itself, with entropy of 0.18 bits. Head 5 puts 91.9% there. That means those two heads gather almost no information from other token positions for this row. It does not make them no-ops. A self-position value can be transformed, the head output can be changed by its projection, and the residual connection can preserve or combine the old state. Heads 0, 2, 9, 10, and 11 all give their largest coefficient to “The.” Head 6 is the only head here whose largest non-self coefficient reaches “France.” That is a statement about these twelve rows on this input, not a classification of what the heads learned across other inputs.

That pile of heads staring at “The” bothered me, so I chased it, and it’s the position-0 curiosity coming back. Here’s how much of the last token’s attention lands on position 0, averaged across all twelve heads, at every layer of the network:

attention on position 0 ('The'), mean over heads, per layer:
  layer 0:  29.2%     layer 4:  53.8%     layer 8:  77.2%
  layer 1:  53.0%     layer 5:  73.2%     layer 9:  79.5%
  layer 2:  48.1%     layer 6:  74.4%     layer 10: 82.1%
  layer 3:  55.0%     layer 7:  88.3%     layer 11: 66.5%

The coefficient grows to 88.3% at layer 7. Calling that a harmless drain would require knowing what happens when the first position changes. I tested six short prompts. Across them, the last row sent a mean 63.50% of its attention to position 0. I then made two interventions and compared the final probability distribution with the unmodified run:

six prompts, final distribution relative to the original:
  mean attention coefficient on position 0:           63.50%
  mask position 0 from every attention row:  mean KL = 4.7641
  replace only the first token:              mean KL = 0.1615

opening prompt only:
  attention coefficient on position 0:                65.02%
  mask position 0:                           KL = 4.9308
  replace "The" with another first token:     KL = 0.1863

The first intervention is violent. Masking a column removes the token’s content, prevents every head from reaching that position, and renormalizes all remaining softmax coefficients. Its large divergence says position 0 cannot simply be deleted from this computation. It does not isolate whether the model needed the value stored there or the normalization pattern created around it. Replacing the first token preserves a usable position but changes its token embedding. That smaller divergence is consistent with the position acting differently from an ordinary content word, but it still does not prove a null operation.

The StreamingLLM paper names the strong allocation of attention to initial tokens an attention sink. Its experiments concern windowed generation and show that retaining a few initial tokens stabilizes models after most older tokens are evicted. My short GPT-2 prompts show the coefficient pattern associated with that result. They do not reproduce StreamingLLM’s long-context serving experiment. The position embedding norm of 9.88 is another correlation. Nothing here shows that the large norm caused the large coefficient, so I cannot turn the two observations into one learned design decision.

the old row had to survive the new one

Attention is only one update to a token’s state. GPT-2’s implementation uses two other operations that are easy to miss because neither moves information between token positions. The first is layer normalization. For one row, it subtracts the row’s mean and divides by its standard deviation, with a small epsilon to avoid division by zero. Learned scale and offset vectors then let each coordinate change the normalized result. Leaving those learned vectors at scale 1 and offset 0 gives this small calculation:

input row:                  [1, 2, 3]
mean:                        2
mean squared deviation:      2/3
layer norm output:          [-1.224736, 0, 1.224736]

The absolute level disappeared, but the relative ordering remained. GPT-2 applies this independently to every token row. It does not compute one mean across the sentence.

The three-number result can be reconstructed. The centered row is [-1, 0, 1]. Squaring and averaging gives variance (1 + 0 + 1) / 3 = 2/3. Transformers 4.57.1 loads this GPT-2 configuration with epsilon 0.00001, so the divisor is

sqrt(2/3 + 0.00001) = 0.816503...

Dividing the centered values by it gives approximately [-1.224736, 0, 1.224736]. The implementation uses the population variance across all 768 coordinates in one token row. It does not use the sample-variance correction that would divide by N-1.

Each layer normalization also owns 768 learned scale values and 768 learned offsets. If normalized coordinate i is z[i], its final result is

scale[i] × z[i] + offset[i]

The small example uses scale 1 and offset 0. The loaded parameters need not stay there after training. This affine step means normalization constrains the raw row statistics without permanently forcing every output coordinate to unit scale. The model can restore, suppress, or shift individual directions.

Epsilon is not an arbitrary accuracy tweak. If every coordinate in a row is equal, every centered value is zero and the variance is zero. Division by sqrt(0) would be undefined. Adding a positive epsilon keeps the denominator finite. Changing epsilon changes the numerical function, especially for nearly constant rows, so it is part of the pinned model configuration.

The second operation is ordinary addition. If a block computes an update [0.25, -0.5, 1] for state [1, 2, 3], the result is [1.25, 1.5, 4]. This preserved route around a sublayer is the residual connection. In the exact Transformers 4.57.1 GPT-2 implementation, one block can be written schematically as:

a = x + attention(layer_norm_1(x))
y = a + mlp(layer_norm_2(a))

This is a pre-normalization arrangement because normalization occurs before each sublayer. The addition explains why a head that mostly attends to itself is not the whole state transition. Attention produces an update. The earlier state remains on the other side of the plus sign. The same structure also gives the model a continuous object to revise through twelve blocks. That object is the residual stream.

one block left every shape visible

The schematic equation could still hide a mismatched transpose or a framework operation doing more than the prose claims. I reconstructed block zero from its modules and compared the result with calling the complete block. The first dimension below is the batch. This run has one input sequence, so it stays 1. The second dimension is token position. It stays 5 until attention creates a pairwise score grid. The last dimensions hold representations:

input ids             (1, 5)
token rows            (1, 5, 768)
position rows         (1, 5, 768)
initial residual      (1, 5, 768)

combined QKV          (1, 5, 2304)
query heads           (1, 12, 5, 64)
key heads             (1, 12, 5, 64)
value heads           (1, 12, 5, 64)
attention weights     (1, 12, 5, 5)
attention update      (1, 5, 768)
after attention       (1, 5, 768)

MLP wide              (1, 5, 3072)
MLP activated         (1, 5, 3072)
MLP update            (1, 5, 768)
block output          (1, 5, 768)

The combined QKV row has width 2,304 because one learned projection computes three width-768 rows at once. Splitting the last dimension into three pieces recovers query, key, and value. Reshaping 768 as 12 × 64 exposes the head dimension. No data is added by the reshape. It changes how the same 3,840 numbers are indexed.

For each head, a (5, 64) query grid multiplies a transposed (64, 5) key grid. The result is (5, 5). Twelve heads and one batch make the complete coefficient shape (1, 12, 5, 5). Multiplying each coefficient row by its (5, 64) value grid returns (5, 64) per head. Concatenating twelve heads restores width 768.

This shape arithmetic catches a common mistake about attention cost. The Q, K, and V projections each scale roughly with T × D², where T is token count and D is model width. The pairwise score and value products scale with T² × D after all heads are counted. Heads redistribute the width; they do not multiply the leading total by twelve again:

projection work:       approximately 3 × 2 × T × D × D operations
score work QK^T:       approximately     2 × T × T × D operations
value blend:           approximately     2 × T × T × D operations
output projection:     approximately     2 × T × D × D operations

The factors of two count a multiplication and addition. Bias operations, softmax, masks, normalization, and memory movement are omitted, so this is a model rather than an exact instruction count. It still explains the change with sequence length. At short T, the width-squared projections can dominate. As T grows, the two pairwise terms grow quadratically.

The MLP never creates a token-pair dimension. It expands each (768) row to (3072), applies GELU coordinate by coordinate, and projects the row back to 768. Its two matrix products cost approximately

2 × T × 768 × 3072
+
2 × T × 3072 × 768
= 4 × T × 768 × 3072 operations

For five tokens that is about 47.2 million operations in one block’s MLP. The four attention projections contribute about 23.6 million. The two attention pair products contribute only 4 × 5 × 5 × 768 = 76,800 operations. On this five-token prompt, the famous quadratic term is tiny compared with the dense projections. At a much longer context, the balance changes.

The manual sequence was exactly:

normalized_1 = block.ln_1(initial)
attention_update, weights = block.attn(
    normalized_1,
    output_attentions=True,
)
after_attention = initial + attention_update

normalized_2 = block.ln_2(after_attention)
wide = block.mlp.c_fc(normalized_2)
activated = block.mlp.act(wide)
mlp_update = block.mlp.c_proj(activated)
reconstructed = after_attention + mlp_update

Dropout is inactive because the loaded model is in evaluation mode. The maximum absolute difference between reconstructed and block(initial)[0] was exactly 0 in this run. That equality is a source-level and execution-level check on the residual equation. It does not mean floating point addition is exact in general. Both paths called the same kernels in the same order.

The last token’s vector norms changed like this:

initial residual:       4.9377
attention update:      28.3575
after addition:        29.2211
MLP update:            32.8798
block output:          56.5040

A larger update norm does not imply that the sublayer erased the input. Vectors have direction. Two large vectors pointing in opposite directions can produce a small sum, while aligned vectors reinforce. Norm also does not identify which output logit will change, because the final projection depends on direction relative to 50,257 output rows. The table only confirms that neither update is numerically negligible for this token in block zero.

Where the answer actually shows up

GPT-2 stacks twelve of those blocks. A diagnostic called the logit lens takes an intermediate residual row, applies the model’s final layer normalization, then projects it through the output weight. GPT-2 ties that output weight to its input token embedding table, so the same 50,257 by 768 parameters are used in both directions. The lens asks what distribution that final readout would produce if it were attached at an earlier block. That distribution is not one the normal forward pass emits. The later blocks still receive the residual vector, not the early lens probabilities.

I did that for the last token, “is,” at every block, tracking the top guess and the zero-based rank of “Paris.” Rank 0 means first, rank 1 means second:

layer   top-1 token   p(top1)   rank of Paris   p(Paris)
  0        ' not'       33.2%       16523          0.000%
  1        ' now'       27.1%       18482          0.000%
  2        ' now'       36.6%       13623          0.000%
  3        ' now'       51.5%        8648          0.000%
  4        ' now'       62.7%        3105          0.000%
  5        ' now'       55.2%        3039          0.000%
  6        ' now'       57.2%         199          0.006%
  7        ' now'       68.9%         274          0.002%
  8        ' now'       42.3%          51          0.062%
  9     ' France'       62.1%           1         18.195%
 10     ' France'       24.2%           1         13.915%
 11       ' the'        8.5%           4          3.225%

Through block 5, “Paris” is below rank 3,000. At block 8 it reaches rank 51. At blocks 9 and 10 it is rank 1, which makes it the second token, not the first. ” France” remains above it. The final block puts “Paris” at rank 4 and 3.225%. That last row reproduces the opening distribution and checks the lens implementation. The earlier rows are probes attached to states that were not trained to be final outputs. They show that the output direction associated with “Paris” becomes much stronger, then weaker. They do not show that an intermediate block contains a stable fact or that the final block reconsidered it.

One capital could be an accident, so I repeated the same ranking probe for five prompts:

prompt ending       best intermediate block and rank       final rank
France is           block 9:  Paris rank 1, 18.195%        rank 4,  3.225%
Japan is            block 10: Tokyo rank 0, 46.114%        rank 1,  6.727%
Germany is          block 10: Berlin rank 0, 34.657%       rank 1,  5.273%
Italy is            block 11: Rome rank 0, 15.735%         rank 0, 15.735%
Canada is           block 10: Ottawa rank 2,  3.064%       rank 22, 0.525%

The shape varies. Tokyo and Berlin become the top lens token one block before the end, Rome is strongest only at the actual output, Ottawa never rises above third and falls to rank 22, and Paris remains behind France. The repeated result is narrower than my first story: some answer-token directions are stronger before the final block on these prompts. There is no universal “the answer appears at layer 9” rule here.

That’s a satisfying story and I don’t fully trust it, because it’s the kind of story that sounds good and could be completely wrong, so I built a test the hedging hypothesis has to pass. If the model is hedging because a bare “The capital of France is” invites a hundred grammatical continuations, then removing that ambiguity should make it commit. So I gave it a prompt that establishes the pattern first, “The capital of France is Paris. The capital of Japan is”, where the only sane continuation is the bare answer, and measured how spread out the output distribution is using its entropy (in bits, where 0 is total certainty and the maximum for 50,257 options is 15.6 bits):

prompt                                              entropy   top-1     90% of mass
'The capital of France is'                          8.65 bits ' the' 8.5% 1503 tokens
'The capital of France is Paris. The capital of     3.09 bits ' Tokyo' 61% 17 tokens
     Japan is'

The second prompt has lower entropy and puts 61.1% on Tokyo. It changes more than answer format: it adds a demonstration, a country, a capital, punctuation, and nine earlier tokens. The result shows that this intervention makes the distribution narrower. It does not isolate which change caused that narrowing, and it cannot establish what the model “always knew.” The original five-token prompt remains a next-token distribution, not a geography answer API.

Two other prompts make the same warning useful. '2 + 2 =' gives 3 at 10.2% as its top guess. The tokenizer represents the expression as tokens, but that alone cannot explain the error. Training examples, model size, the next-token objective, and the prompt are all entangled. 'The quick brown' puts 'ie' at 19.2%, a continuation toward “brownie” rather than the pangram. That is direct evidence about these output distributions. Assigning either failure to one internal cause would need an intervention that changes that cause while holding the others fixed.

Two-thirds of the model is the part I haven’t mentioned

I have spent this entire post on attention. Twelve heads, the sink, the KV cache, the query-key-value dance, all of it attention, because attention is the mechanism people write blog posts about and it’s genuinely the clever part. But if you ask where GPT-2’s 124 million parameters actually are, attention turns out to be the smallest computational block in the network:

where GPT-2's 124,439,808 parameters live:
  MLP (c_fc + c_proj)        56,669,184   45.5%
  token embeddings (wte)     38,597,376   31.0%
  attention (c_attn+c_proj)  28,348,416   22.8%
  position embeddings (wpe)     786,432    0.6%

Attention is 22.8% of the weights. The MLP, the feed-forward layer I’ve mentioned twice in passing and never explained, is 45.5%, twice as big, and the embedding table (input and output, since they’re tied) is another 31%. So the parts I obsessed over are a fifth of the model, and the two parts I waved at (a plain feed-forward network and a lookup table) are three-quarters of it. That’s worth sitting with, because it’s easy to leave a post like this thinking a transformer is “mostly attention.” It is mostly a stack of ordinary two-layer neural networks with attention sprinkled between them to shuffle information across positions.

Those percentages came from counting the loaded parameters, but the count can also be rebuilt from the shapes. Block zero contains:

attention QKV:
  768 × 2304 weights + 2304 biases      = 1,771,776

attention output:
  768 × 768 weights + 768 biases        =   590,592

MLP up:
  768 × 3072 weights + 3072 biases      = 2,362,368

MLP down:
  3072 × 768 weights + 768 biases       = 2,360,064

two layer norms:
  two scales and two offsets, width 768 =     3,072

one block total                          = 7,087,872

Multiplying by twelve gives 85,054,464 block parameters. The rest are:

token table:       50,257 × 768 = 38,597,376
position table:     1,024 × 768 =    786,432
final layer norm:    scale + offset =      1,536

total unique parameters            = 124,439,808

There is no additional 50,257 × 768 output table in that total. The language-model head points to the token table’s storage. I checked the two tensors’ data pointers and they are equal. Counting modules without accounting for that tie would add 38,597,376 parameters that do not exist.

Parameter count is not activation memory. The weights persist across requests. Intermediate rows depend on batch and sequence length. For batch B and tokens T, the MLP’s wide fp32 activation alone contains B × T × 3072 × 4 bytes. At B=1, T=5, that is only 61,440 bytes. At B=8, T=1000, it is 98,304,000 bytes, about 93.75 MiB, before other live tensors and allocator workspace. In inference, implementations can release or reuse intermediates once their consumers finish. During training, backward computation often needs saved activations, which changes the memory problem.

Each block runs every token’s 768-dimensional row through the same MLP: project to 3,072 dimensions with c_fc, apply GELU, then project back to 768 with c_proj. Every position shares the matrices, but each has different input values. The expansion provides 3,072 coordinates on which a nonlinear function can act.

Without a nonlinearity, two consecutive matrix products collapse into one matrix product. If y = (xA)B, associativity gives y = x(AB). Adding more such linear layers would not create a new class of transformation. GELU breaks that collapse because it changes each intermediate value according to its sign and magnitude before the second matrix sees it.

GPT-2 uses the approximate GELU form configured as gelu_new. For an input x, it is approximately:

0.5 × x × (1 + tanh(sqrt(2/pi) × (x + 0.044715 × x³)))

At a large positive input the parenthesized factor approaches 2, so the output approaches x. At a large negative input it approaches 0, so the output approaches 0 from the negative side. Near zero, values change smoothly rather than hitting the hard corner of max(0, x). I inspected the 3,072 post-GELU values for token “is” in block zero:

MLP hidden activation (layer 0, token 'is'), 3072 wide:
  fraction ~zero (|x| < 0.01):  2.9%
  fraction negative:           90.6%
  max activation:  3.65   min: -0.17

It is not sparse by that threshold. Only 2.9% is within 0.01 of zero, while 90.6% is negative. GELU differs from a hard ReLU because it can retain a small negative result instead of replacing every negative input with zero. This one activation row contains many small negative values and a few larger positive ones reaching 3.65. I have not identified what any coordinate computes, and parameter count cannot tell me where the model stores a fact. What it does establish is structural: the MLP owns 45.5% of GPT-2’s parameters and transforms each token row separately. Attention moves information among positions. The MLP changes the representation at one position. The residual addition then preserves both routes for the next block.

The cache that makes generation possible

So far I’ve described processing five tokens once. But generating text means producing tokens one at a time: predict the sixth token, append it, predict the seventh, append it, and so on. The naive way to do that is to re-run the whole forward pass over the whole growing sequence every time, which is quadratically wasteful, token 100 would re-process tokens 1 through 99 from scratch.

The fix is the KV cache. Causal attention means adding a later token does not change the key and value rows already computed for earlier positions. The implementation can retain those rows and compute only one new key and one new value per layer. Generation then has two phases. Prefill processes all prompt rows and constructs the initial cache. Decode supplies one new input row and returns one new cache row per layer.

first cached key tensor:
  shape = (batch 1, heads 12, sequence 16, head width 64)
  dtype = fp32

There is one key tensor and one value tensor at each of twelve layers. The byte calculation for one cached token is therefore:

2 tensors
× 12 layers
× 12 heads
× 64 values per head
× 4 bytes per fp32 value
= 73,728 bytes per token

The returned cache for sixteen tokens occupied 1,179,648 bytes, exactly sixteen times that result. My earlier 36,864 byte number silently assumed fp16 even though this CPU run returned fp32. At fp16 the formula does produce 36,864 bytes per token. At this run’s fp32, a full 1,024-token GPT-2 cache is 72 MiB.

The cache belongs to a sequence, not merely to the model. With active sequence lengths L1 through LN, total live GPT-2 cache payload is:

73,728 bytes × (L1 + L2 + ... + LN)

The batch-8 timing at context 64 therefore had a derived live cache payload of

73,728 × 8 × 64 = 37,748,736 bytes = 36 MiB

If those eight requests each reached 1,000 cached tokens, the payload would be 562.5 MiB. Model weights are shared across the batch; cache rows are specific to each request’s token history. This is why increasing batch can amortize weight use while still increasing memory consumption.

A simple server can reserve the maximum context for every admitted request. With 100 slots and a 1,024-token maximum, this fp32 GPT-2 cache would reserve 7,200 MiB even if most requests contain only a few tokens. Allocating only exact current lengths avoids that reservation but requires frequent growth and movement. A paged cache splits storage into fixed-size blocks and maps a sequence’s logical token positions onto whatever physical blocks are free. The logical order stays visible through the block table even when physical blocks are not adjacent.

Paging changes allocation, not attention’s mathematical work. A query still needs every key and value in its logical context. The kernel now follows block mappings or consumes a gathered view. Smaller blocks reduce unused space at the end of a sequence but enlarge mapping metadata and can make memory access less regular. Larger blocks reduce mapping overhead but waste more capacity when a request ends partway through one.

Prefix sharing adds another ownership problem. Two requests with identical initial ids can reference the same immutable cache blocks for that prefix. Once their generated ids differ, later blocks must be distinct. Eviction also cannot remove a shared block until every referencing request has released it. These are serving policies layered on the key and value tensors measured here. They do not change the per-token byte formula, but they determine how much of the allocated capacity holds useful cache rows.

The shape also exposes two ways larger models change the result. A hypothetical model with 32 layers, width 4,096, 32 key and value heads, and fp16 cache values needs 2 × 32 × 32 × 128 × 2 = 524,288 bytes per token. That is 1 GiB at 2,048 tokens and 4 GiB at 8,192 tokens. Those are derived dimensions, not a measurement of a named model.

Grouped-query attention reduces the number of key and value heads while retaining more query heads. Multi-query attention uses one key and value head per layer. If the hypothetical model used eight key and value heads instead of 32, its cache would be one quarter as large. GPT-2 does not do this. Its twelve query heads have twelve key heads and twelve value heads. Nor does every later model use grouping. The exact architecture has to be inspected before using the smaller formula.

The cache optimization should preserve the logits that full-prefix recomputation produces. I tested that invariant for five greedy steps. The uncached path appends the chosen id and runs the entire growing sequence. The cached path supplies only that new id together with the cache returned by the previous call:

prefix = tokenizer("The capital of France is")
first = model(prefix, use_cache=True)
cache = first.past_key_values
next_id = argmax(first.logits[:, -1])

for step in range(5):
    full_prefix = append(full_prefix, next_id)

    cached = model(
        next_id,
        past_key_values=cache,
        use_cache=True,
    )
    complete = model(full_prefix, use_cache=False)

    compare(cached.logits[:, -1], complete.logits[:, -1])
    cache = cached.past_key_values
    next_id = argmax(complete.logits[:, -1])

The real probe uses tensors rather than the abbreviated append, argmax, and compare names. It produced:

step   new input     maximum logit difference   mean difference   same top id   cache length
  1    ' the'             0.00021362              0.00003452         yes             6
  2    ' capital'         0.00016785              0.00003660         yes             7
  3    ' of'              0.00018311              0.00003701         yes             8
  4    ' the'             0.00021362              0.00005088         yes             9
  5    ' French'          0.00019073              0.00003775         yes            10

The logits are not bit-identical. Cached decode multiplies one query row by a stored key grid, while full recomputation processes a larger set of rows. Kernel selection and floating point grouping can differ even when the mathematical expressions are equivalent. The largest observed absolute difference is about 2.14 × 10^-4, and the top token id agrees at all five tested steps. That validates this short path at the decision boundary used by greedy decoding. It does not guarantee agreement indefinitely. If two logits are nearly tied, a smaller numerical difference could change the selected id, after which the generated prefixes diverge.

The cache length also gives a useful state invariant. The first call sees five ids and returns length five. Each cached call consumes one new id and increases the length by exactly one. Supplying the full prefix together with that same cache would duplicate old positions. Supplying the wrong position id would combine a new token with the position embedding for another location. A serving engine has to keep token ids, cache length, position indices, and attention masks consistent for every active request.

The cache removes repeated projection of old rows, but it cannot make decode independent of context length. A new query must still be compared with every cached key, and its result blends every cached value. That work is linear in the number of cached positions. I measured seven randomized rounds at each context length and report the median, with the 10th and 90th percentiles in parentheses:

per-token decode time vs context length:
  ctx=   8:  8.84 ms  (8.75 to 9.04)
  ctx= 128: 10.35 ms  (9.44 to 12.46)
  ctx= 512: 11.10 ms  (10.99 to 11.49)
  ctx=1000: 13.24 ms  (13.00 to 13.27)

From 8 to 1,000 cached positions the median rises by 50%. I cannot assign all of that change to one kernel without a profiler trace. The direction and the linear attention term agree, but allocator behavior, framework dispatch, cache hierarchy, and CPU scheduling are not controlled away. The narrower statement survives: the cache avoids recomputing old projections, while both its storage and the new query’s attention work grow with context.

Why the tokens come out slowly

The profiling investigation treated a fast run as evidence that needed a mechanism. The same restraint matters here. GPT-2 has 124,439,808 parameters. A rough dense-layer count gives about two arithmetic operations per parameter for one sequence row, one multiplication and one addition. If every fp32 weight had to arrive from DRAM once for that row, the arithmetic intensity would be about:

(2 operations × 124,439,808 parameters)
/
(4 bytes × 124,439,808 parameters)
= 0.5 operations per byte

That is a model, not a counter reading. Weights can remain in caches, operators add work that is not proportional to parameter count, and the runtime can fuse or copy tensors. A memory roofline would multiply available bytes per second by 0.5 operations per byte and compare that ceiling with the processor’s arithmetic ceiling. I have neither reliable hardware counters nor an instrumented memory controller for this run, so I cannot state that the measured decode is memory bound.

Batching gives the model a useful falsification test. If each request read an entirely separate copy of every weight, doubling the batch would double step time and total tokens per second would remain flat. If a matrix operation reuses a weight across rows, step time can grow more slowly than the batch. I filled each request to context 64, decoded one next-token row, randomized the order, and ran seven rounds:

batch   median step     total tokens/s   parameter-byte normalization
  1       10.06 ms           99.4                 49.5 GB/s
  2       15.23 ms          131.3                 65.3 GB/s
  4       15.51 ms          257.8                128.3 GB/s
  8       17.66 ms          453.1                225.5 GB/s

The final column is tokens/s × 124,439,808 × 4. It is deliberately called a parameter-byte normalization, not measured DRAM bandwidth. At batch 8 it reaches 225.5 GB/s precisely because the same weight can serve several rows after it has been fetched into the execution path. Treating that column as physical traffic would produce an absurd conclusion. My old calculation compared such a normalization with a NumPy sum and called the ratio proof of streamed DRAM bytes. It was not.

The actual observation is stronger in a different direction. Moving from batch 1 to batch 8 raises median step latency from 10.06 to 17.66 ms, only 1.76 times, while total throughput rises 4.56 times. The idealized arithmetic intensity rises from about B/2, or 0.5 operations per byte at batch 1, to 4 operations per byte at batch 8 when one weight fetch serves eight rows. This is why decode batching can improve hardware use. It also makes each request wait for a larger step and creates a scheduling problem when requests arrive at different times.

Prefill has another form of row reuse. I measured full prompt passes using the same randomized seven-round protocol:

context   median prefill   10th to 90th percentile   aggregate tokens/s
    8       15.99 ms          15.78 to 16.11               500.3
   32       26.71 ms          25.82 to 29.16             1,198.1
  128       57.73 ms          57.13 to 58.56             2,217.1
  512      188.04 ms         182.29 to 192.95            2,722.8
 1000      343.84 ms         341.22 to 349.08            2,908.3

A thousand-token prefill does not cost 125 times the eight-token run. It exposes far more parallel row work to the matrix kernels, even while causal attention adds work as the sequence grows. These CPU timings do not prove that prefill is compute bound, and they say nothing directly about a GPU. They show the operational split that a serving scheduler sees: prefill presents many prompt rows together, while an unbatched decode step presents one new row and a growing cache.

The full verifier ran the timing process again after the other probes. Its prefill medians were 15.27, 24.39, 52.66, 184.53, and 344.23 ms for the same five lengths. The 32-token and 128-token medians shifted by almost 9%, while the 1,000-token median shifted by 0.1%. Decode at context 1,000 changed from 13.24 to 12.90 ms. Batch-8 step time changed from 17.66 to 17.57 ms.

That repeat is a warning about treating one short timing table as a hardware constant. The probe randomizes case order within a process and reports seven rounds, but it does not fix CPU frequency, pin the process to cores, flush caches, or isolate other operating-system work. Longer cases are more stable here because a brief disturbance is a smaller fraction of the run. I retained the first complete run above and recorded the repeat instead of silently replacing it with the nicer numbers.

the integers were never packed

Here is the up-projection weight from GPT-2’s first MLP:

weight c_fc: shape (768, 3072), fp32, 2,359,296 numbers, 9216 KiB
sample fp32 values: [0.0942, 0.0982, -0.0321, 0.0671, -0.1154, 0.0026]

Each number is a 32-bit float, four bytes. A symmetric group quantizer chooses one scale for 64 adjacent weights, rounds each ratio weight / scale to one of the integer codes from negative 8 through 7, then reconstructs an approximation as code × scale. I applied that rounding:

4-bit codes, same 6 weights:   [2, 2, -1, 1, -3, 0]
scale for their group:          0.04576
theoretical packed payload:     1224 KiB
original fp32 payload:          9216 KiB
executed approximation dtype:   fp32

The weight 0.0942 became code 2, which reconstructs as 2 × 0.04576 = 0.09152. Sixty-four four-bit codes would need 32 packed bytes. One fp16 scale adds 2 bytes, for 34 bytes per group instead of 64 × 4 = 256 fp32 bytes. That is where the 7.53 ratio comes from.

The probe did not create those 34 bytes. It wrote the reconstructed approximations back into ordinary fp32 PyTorch parameters. The model still occupied the fp32 execution payload and ran fp32 matrix operations. Therefore this experiment can measure numerical error. It cannot measure packed storage, dequantization overhead, memory traffic, or speed. The payload column is arithmetic describing a format that a real packed implementation could use.

Applied to seven billion parameters, the same format would require about 3.5 decimal GB for codes plus 0.219 GB for fp16 scales:

7B parameters:
  fp32 payload:                 28.000 GB
  fp16 payload:                 14.000 GB
  4-bit codes + fp16 scales:    3.719 GB

That is only the selected weight payload. A process also needs unquantized parameters, temporary activations, a cache, runtime workspaces, and allocator overhead. Fitting 3.719 GB of codes does not prove that the model process fits in the same capacity. Reading one quarter as many weight bytes could raise a memory-limited kernel’s ceiling, but code unpacking, scale reads, conversion, arithmetic throughput, and incomplete weight coverage prevent a fourfold speed claim without a packed kernel measurement.

The approximation can still be tested. I rounded every large two-dimensional weight parameter this way, reconstructed fp32 parameters, and reran the exact prompt:

fp32 top-5:   ' the' 8.5%   ' now' 4.8%   ' a' 4.6%   ' France' 3.2%   ' Paris' 3.2%
4-bit top-5:  ' a'  41.2%   ' the' 33.1%  '.'  12.7%  ','     0.9%   ' and'  0.8%

France and Paris left the top five, while a received 41.2%. The mean absolute reconstruction error in the sampled MLP matrix was 0.01453. Its weight standard deviation was 0.1412, so the error was 10.3% of that scale. That does not mean every individual weight has 10.3% relative error, since values near zero make such a ratio unstable. It says the approximation is large relative to the spread of this matrix.

“Noticeably dumber” is a soft word for something I can put a number on, so let me. If I take the fp32 output distribution as ground truth and measure how far the quantized distribution has drifted from it (the KL divergence, which is 0 for identical distributions and grows as they diverge), I can sweep the number of bits and watch exactly where the model falls off a cliff:

bits   KL(fp32 || quant)   top-1    p(top1)   p(Paris)
  8         0.0166         ' the'     6.0%      3.869%
  6         0.4314         ' the'    30.6%      1.605%
  4         4.8316         ' a'      41.2%      0.175%
  3        15.5664         ' an'     28.2%      0.000%
  2        13.2413         ','       48.5%      0.000%

On this prompt, 8-bit rounding gives KL 0.0166 and preserves the top token. Four-bit rounding gives KL 4.8316 and changes it. KL is asymmetric and this is only one input distribution, so the table does not establish a general quality threshold at four bits. It establishes that this quantizer and this prompt are sensitive somewhere between the tested six-bit and four-bit settings.

Since the crude version is standing on a cliff edge, the whole game is buying back margin, and I wanted to know which knob buys the most. First knob, the group size. My crude version shares one scale across 64 weights; what if I share it across fewer, so the scale fits each little neighborhood better?

group size   KL      top-1    p(Paris)   (all at 4-bit)
    16      1.6072   ' the'    0.338%
    32      4.2785   ' the'    0.196%
    64      4.8316   ' a'      0.175%
   128      7.1826   ' a'      0.003%
  1024     10.7789   ' just'   0.000%

On the same prompt, reducing the group from 64 weights to 16 lowers KL from 4.8316 to 1.6072. A smaller group can fit its scale to fewer values, but it stores more scales. With fp16 scales, group 64 costs 0.5 + 2/64 = 0.53125 bytes per weight. Group 16 costs 0.5 + 2/16 = 0.625 bytes per weight. The table is one observed quality curve plus a derived storage curve, not a universal optimum.

I then rounded one parameter family at a time, leaving every other parameter at its original value:

only this quantized to 4-bit    KL(fp32 || quant)
  attn.c_attn (Q,K,V)  ×12          0.0709
  attn.c_proj          ×12          0.0466
  mlp.c_fc  (up)       ×12          0.0400
  mlp.c_proj (down)    ×12          0.0449
  wte (embeddings)     ×1           4.2970

The embedding table is the most sensitive family for this prompt. GPT-2 ties it to the output projection, so changing it changes both the input lookup and every final logit. This experiment does not isolate which use causes how much damage. It also compares each family separately, not all 48 compute matrices at once. The correct result is simply that the one-family wte intervention gives KL 4.2970, while each tested twelve-matrix family gives KL between 0.0400 and 0.0709 on this output.

Keeping token and position embeddings in fp32 while rounding the other large matrices with group 32 makes the opening distribution closer:

careful 4-bit:  ' the' 12.1%   ' a' 4.6%   ' now' 3.7%   ' in' 2.9%   ' France' 2.2%

The KL falls from 4.8316 to 0.1542 and the is top again. The price is large because the tied token table contains 38,597,376 parameters, 31.0% of the model. The whole fp32 parameter payload is 474.70 MiB. Across the complete model, four-bit codes with fp16 scales for all large matrices would have a theoretical payload of 63.45 MiB, a 7.48 ratio. Keeping embeddings in fp32 raises it to 196.26 MiB, only a 2.42 ratio. Preserving the sensitive table is not free.

There’s one more reason the crude version bleeds, and it’s hiding in the weights themselves. My per-group scale is set by the largest weight in each group of 64, because the scale has to be big enough to represent the biggest value. So I went looking at the distribution of that c_fc matrix:

c_fc:  std = 0.1412,  max |weight| = 4.5877  = 32.5 sigma
       fraction of weights beyond 4 sigma: 0.126%

The standard deviation is 0.1412, not the typical absolute weight. The largest absolute value is 32.5 times that standard deviation. If one group contains such a value, a maximum-based scale must cover it, making the step coarser for the smaller values in that group. The measured 0.126% beyond four standard deviations describes this matrix as a whole. I did not count how many distinct groups they affect, and I did not implement a separate outlier path, so “poisoning every group” would go beyond the evidence.

one sentence was not an evaluation

KL against the fp32 output says how much one predicted distribution moved. It does not say whether the changed model became more or less accurate, and a single capital prompt is an especially poor evaluation set. I added a fixed local passage containing 273 next-token prediction positions. It mixes the capital sentences with prose about files, replication, prefill, decode, batching, and quantization. It is still tiny and written from the same material as this site, but it can expose a change that one final token hides.

For each position I measured the negative log probability assigned to the actual next token. Averaging those values gives negative log likelihood. Exponentiating it gives perplexity, which can be read as an effective number of equally plausible choices. Lower is better on this fixed passage. I also measured mean KL from the fp32 distribution and the fraction of positions where the rounded model chose the same top token:

case                                      payload   ratio   perplexity   mean KL   top-1 agreement
fp32                                      474.70 MiB  1.00     119.635     0          100.00%
8-bit, all large matrices, group 64       122.73 MiB  3.87     121.991     0.0146      90.48%
4-bit, all large matrices, group 64        63.45 MiB  7.48   6,236.476     4.2950      26.37%
4-bit compute, fp32 embeddings, group 32  196.26 MiB  2.42     145.831     0.2262      75.82%

The payloads remain theoretical and execution remains reconstructed fp32. The 8-bit approximation changes the top choice at about one position in ten while moving perplexity by 2.0%. That is small on this passage, not “free.” The crude four-bit approximation fails badly across the passage. Keeping embeddings restores much of the numerical behavior, but perplexity remains 21.9% above fp32 and one top choice in four changes.

A useful packed implementation would need at least three additional experiments. It would store the codes rather than reconstructed weights, run a kernel that reads codes and scales, and compare task or corpus quality on representative data. Only then could reduced payload, kernel time, and model behavior be placed on the same curve.

the batch changed the price and the wait

Throughput becomes a cost only after attaching units. If a machine costs H dollars per occupied hour and produces T output tokens per second, then its machine cost per million output tokens is

H dollars     1 second        1,000,000 tokens
---------  ×  ---------  ×  ----------------
  hour        T tokens

= H × 1,000,000 / (3,600 × T) dollars

The seconds cancel and the tokens cancel, leaving dollars. At an explicitly assumed machine price of $1 per hour, the batch 1 throughput of 99.4 tokens per second gives $2.79 per million output tokens. The batch 8 throughput of 453.1 gives $0.61. These are substitutions into the formula, not the operating cost of this laptop and not a cloud price.

The cheaper number changes latency. One batch 1 step took a median 10.06 ms. One batch 8 step took 17.66 ms. A real server also waits to assemble work, removes completed requests, admits new requests, processes prompt prefill, and may pad or split uneven sequences. Total tokens per second can improve while one request waits longer. Cost cannot select a batch policy until a latency target is specified.

Nor is machine rental the complete numerator. Power, host memory, networking, idle capacity, failed requests, replicas, engineering work, and the fraction of time the process is usefully occupied all matter. If only fraction U of paid time produces output tokens, effective throughput in the cost equation is U × T. At 50% useful occupancy, the machine component doubles. A fourfold smaller theoretical weight payload cannot be entered as a fourfold cost reduction. The packed kernel, quality target, batch occupancy, and request latency have to survive together.

Paris was fifth for several different reasons

The measurements in this investigation all use the pinned 124,439,808-parameter GPT-2 snapshot on an Apple M4 CPU. The cache dimensions came from returned tensors. The attention coefficients came from the eager attention implementation. The batching and context tables are wall-clock timings across seven randomized rounds. The quantization results are fp32 executions after simulated rounding, with theoretical packed payloads beside them. I did not run a GPU kernel, measure DRAM transactions, store packed four-bit weights, or test a seven-billion-parameter model.

The first table can now be produced from its execution path. Five token ids select five token and position rows. Twelve residual blocks normalize, mix positions through attention, transform each row through an MLP, and add both updates back into the running state. Final normalization and the tied embedding projection produce 50,257 logits. Stable softmax turns them into the distribution where the is 8.5% and Paris is 3.2%.

The intermediate lens does not show Paris as top for this prompt. It reaches zero-based rank 1, behind France, at blocks 9 and 10. Other capital prompts follow different paths. Masking position 0 changes the distribution too much to call its large attention coefficient a harmless no-op. The cache avoids recomputing old key and value projections, but decode still grows with context. Batching reuses execution work across rows, but increases step latency. Four-bit rounding damages this tiny corpus unless the largest sensitive table remains fp32, and doing that cuts the theoretical compression ratio from 7.48 to 2.42.

Paris was fifth because the model returned a next-token distribution for this exact prompt. Greedy decoding would select the. Sampling could still select Paris. A demonstration prompt changes the distribution. No one line among the tokenizer, attention map, logit lens, cache, or quantizer turns that observation into a single fact stored at a single address. The fifth entry is what remains after all of those rows have been multiplied, normalized, added, and projected once.