A transformer language model, written from scratch in C,
generating in your browser.

The whole thing is one loop

A language model is a function from tokens so far to a probability for every possible next token. Generation just calls it repeatedly: predict, sample, append, repeat. While your prompt is still being fed in, the "next token" is forced to be your prompt; after that, the model is on its own.

A model is a file of floats

The entire model is one flat binary: a 28-byte header of seven ints, then every weight matrix as raw float32, in a fixed order. We mmap it and never copy: the "loaded model" is just pointers into the file.

Laying pointers over the blob

No deserialization. Each weight tensor is found by walking a pointer through the mapped file in the exact order the exporter wrote it. This ordering is the file format. One neat trick: if the header's vocab_size is positive, the final classifier is tied to the embedding table: the same matrix used twice.

One vector, edited by every layer

The spine of a transformer is the residual stream: the token's embedding, which each sub-layer reads, computes a correction to and adds back. Information is never overwritten, only edited. That additive structure is why deep transformers train stably.

Keeping the numbers sane

Before every sub-layer, the stream is rescaled to unit RMS and given a learned per-dimension gain. Five lines. Without it, activations drift in scale as they pass through dozens of matmuls and training falls apart.

Where all the time goes

Every projection in the network (queries, keys, values, outputs, both feed-forward wings, the classifier) is this one function. Each output element is a dot product of one weight row with the input. This loop is ~95% of the runtime; every LLM speed trick you've heard of (quantization, GPU kernels, batching) is fundamentally about making this faster.

Position as rotation

The network has no idea where a token sits; attention is order-blind. RoPE fixes this by rotating each adjacent pair of query/key dimensions by an angle proportional to the token's position (faster for low dims, slower for high: a spectrum of frequencies). The magic: after rotation, the dot product between a query and key depends only on their relative distance.

The only place tokens meet

Everything else operates on each position independently. Attention is where context flows. Each head scores the current token's query against every previous token's key, softmaxes the scores into weights, and takes that weighted blend of their values. Past keys/values sit in the KV cache.

And notice what's missing: there is no causal mask anywhere. A token physically cannot see the future, because the future isn't in the cache yet. The loop bound is the mask.

Where the knowledge lives

After attention gathers context, the feed-forward block does the per-token "thinking": expand to a wider hidden space through two parallel projections, gate one with the other using SiLU, contract back down. Most of the model's parameters (and most of what it knows) are in these three matrices per layer.

259 opinions about what comes next

After the last layer: one final norm, then the residual stream is scored against every row of the (tied) embedding table. The result is a raw score, a logit, for each of the 259 tokens in the vocabulary. That vector is the model's entire output; everything after this is just deciding how to pick from it.

One byte, one token

Real Llama uses 32,000 learned BPE merges. This model goes simpler: the vocabulary is just the 256 possible bytes (plus 3 specials). Nothing to train, and the C encode/decode machinery (including the full BPE merge loop) works unchanged; it simply finds nothing to merge. The price: one byte = one token, so 256 context tokens ≈ 256 characters. The TinyStories model (step 18) swaps in a real 4096-vocab BPE tokenizer: same C code, learned merges instead of none.

Rolling the dice, carefully

Temperature reshapes the distribution, then top-p cuts off the long tail of unlikely tokens before drawing. That's the whole difference between the temp-0 and temp-1.4 output in the demo above. Same model, same logits, different dice.

The same model, written twice

The weights were trained by a PyTorch model that mirrors run.c exactly: same RMSNorm epsilon, same RoPE base and pairing convention, same tied classifier. Get any one of those wrong and the exported model produces fluent garbage: the file loads fine, the math runs fine and every number is subtly wrong. The two implementations agreeing is the correctness proof.

Watching it learn, then memorize

2.26M parameters, ~1.1M tokens of Shakespeare, AdamW, six minutes on a laptop GPU. Loss starts at 5.56 (exactly ln(259), the loss of pure guessing) and the curves below are the real run. Around iteration 750 the validation loss bottoms out and turns back up while training loss keeps falling: the model has started memorizing its corpus instead of learning English. The shipped checkpoint is the one from the bottom of the valley.

The handshake between two worlds

Export is just serialization discipline: write the 7-int header, then dump every tensor in exactly the order step 03's pointer walk reads them. PyTorch stores linear weights as (out, in) row-major, the very layout run.c's matmul indexes, so there's not even a transpose. Train in Python, run anywhere C runs. Which, via WebAssembly, includes the page you're reading.

Four bytes into one

Step 06 said matmul is where all the time goes, but it's not the multiplies, it's streaming the weight bytes from memory. So: store each weight as an int8 plus one fp32 scale per group of 64, quantize the activations on the fly and run the inner loop in pure integer arithmetic. 4× less memory traffic, 4× smaller file. What stays fp32: the norms, the KV cache, everything between ops: precision where it's cheap, bytes saved where it's expensive.

This isn't hypothetical: the engine generating above is runq.c, and the 2.4 MB you downloaded (instead of 9.1) is the quantized checkpoint.

The bug that spoke in tongues

First int8 run of a dim-288 model: fluent English with random exotic tokens sprinkled in: "there was a little girl namedoptsa". Cause: 288 isn't divisible by the group size 64, so quantization groups straddled matrix rows and every odd row read its neighbour's scales. Half the vocabulary's logits corrupted, half perfect: a model that's simultaneously right and wrong. Our Shakespeare model (dims 192/512) masked the bug completely.

The lesson quantized models teach: they fail eloquently. Nothing crashes; the text just gets strange. The fix is one invariant (groups must align with rows), enforced twice:

From letters to words

Try the 📖 TinyStories model at the top. It reads far better than Shakespeare, for two reasons. First, a real byte-pair-encoding tokenizer, trained from scratch: start from the 256 bytes and repeatedly fuse the most common adjacent pair into a new token, to a vocab of 4096. The first merges it learns on children's stories are he,  t,  a, the: the statistical skeleton of English. Now "Once upon a time" is ~4 tokens, not 16, so the same 256-token window reaches four times deeper into a story.

And, the recurring theme of this whole project, run.c needed zero changes. Its encode() was already a greedy "merge the highest-scored pair" loop, which is exactly BPE. We just write the learned merges into the same tokenizer file with score = -rank, and the C engine reproduces our tokenization for free.

Seven million parameters, a real corpus

Second reason: scale. The Shakespeare model saw ~1 MB of text; this one trained on ~300 MB of TinyStories: 79 million tokens, tokenized once into a memory-mapped uint16 array so the GPU never waits on Python. The model itself grew to 7.2M parameters (dim 288, 6 layers), the shape of Karpathy's stories15M but with our leaner 4096 vocab. One epoch is ~4,800 steps, so a few thousand steps of training never even finishes a full pass, the opposite of the Shakespeare model, which memorized its tiny corpus. The result is what you're reading: real words, whole sentences, little arcs with a beginning and a "happily ever after".

Teaching it to follow, not just continue

The base model continues text; the 💬 Instruct model at the top follows an instruction. Same 7M weights, further trained ("supervised fine-tuning") on a chat template built from the stories themselves: pull content words out of a story, and the instruction becomes "write a story using these words," with the story as the target answer:

User: Write a story using the words: dragon, cake, brave.
Assistant: <the story></s>

The one genuinely new idea is loss masking. We do not want the model learning to generate the instruction, only the response. So every target position inside the prompt is set to -100, which PyTorch's cross-entropy silently ignores. The gradient only ever flows from the answer. And, the theme of this whole project once more, run.c is unchanged: the template is plain text, and BOS is still the stop token, so the model just learns to end its turn.

What 7M parameters can and can't do

Set expectations: this is not an assistant. It's a demonstration that the mechanism works. Ask for a story and it obeys the format and theme every time: "about a robot" reliably gets you a robot. Injecting specific requested words is harder: common ones (dog, park, friend) usually land, rare ones often don't. About 24% of random content words make it in (measured on held-out prompts). It has no facts, no reasoning, no memory across turns; it knows only the small world of TinyStories.

Two things it does not use, that real instruct models do: LoRA/PEFT (we full-fine-tune all 7M weights because at this size it's free; large models train a small adapter instead), and RLHF/DPO (a second preference-alignment stage after SFT). SFT is step one, where instruction-following comes from. That 24% is the number the next phase attacks.

Teaching taste without a teacher

SFT taught the model to answer. Alignment teaches it which answer is better. Real RLHF collects pairs where a human picked the better of two responses; we can't hand-label thousands, so we use a programmatic judge, which makes this RLAIF, but the optimizer downstream is identical.

The judge is Phase 6's weakness itself: did the story use the requested words? For each prompt we sample several completions from the SFT model, then pair the one using the most requested words (chosen) against the one using the fewest (rejected). The preferences are the model's own outputs, sorted by a rule it will now be trained to satisfy.

Two models, one nudge

Classic RLHF trains a reward model then optimizes it with RL (PPO). DPO skips both: it optimizes the preference pairs directly. Keep two copies of the SFT model (a trainable policy and a frozen reference) and push the policy to prefer chosen over rejected, but only beyond what the reference already prefers. One line of loss, no reward model, no RL loop.

The result, both numbers measured identically on the same held-out prompts: word-inclusion went from 22% (SFT) to 45% (DPO), roughly doubled; the model learned to actually use the words it's given. The cost, visible in its output, is the alignment tax: reward-hacking that trades fluency for score. Switch between 💬 Instruct and 🎯 Aligned above and give both the same three words. The difference is DPO.

The machinery DPO threw away

DPO was invented to replace this. Before it, alignment meant PPO: reinforcement learning, the InstructGPT recipe. Same reward, same frozen reference, but instead of one offline loss over preference pairs, PPO runs a full RL loop: the policy generates responses on-policy, a value head (critic) scores each state, GAE turns rewards into advantages and a clipped surrogate objective nudges the policy, clipping the probability ratio so no update moves too far (the "proximal" in PPO), with a per-token KL penalty back to the reference.

Five moving parts (rollouts, reward, critic, GAE, clipped+KL loss) regenerated every iteration, versus DPO's one loss over a fixed dataset. That extra machinery is the comparison.

Two roads, same destination

Both start from the SFT model and chase the same reward. On the held-out word-inclusion test: SFT 22% → DPO 45% → PPO ~22%. That's the honest result: our PPO didn't beat the baseline. Not because PPO can't (it's the algorithm behind ChatGPT) but because it's finicky. Stable settings (higher KL penalty) barely moved; aggressive ones (higher LR, lower KL) destabilized: entropy blew up and reward fell. Across three configs, none cleanly beat DPO's robust first-try doubling.

That contrast is the lesson: PPO's high-variance policy gradient vs DPO's low-variance contrastive loss is exactly why DPO became the default for straightforward preference alignment. And the cost gap seals it. PPO needs a critic, GAE, reward shaping, KL/clip tuning and fresh rollouts every iteration (minutes of generation), for a result DPO reached with one loss over a static file. Switch between 🎯 Aligned (DPO) and 🕹️ PPO on the same words. DPO reliably works them in; our PPO, like SFT, often doesn't.

The piece both toys skipped

Why did PPO stall? Two missing pieces, and this phase adds them. First, a real reward model, the middle stage of RLHF that DPO folds away and our PPO faked with the raw rule. We train it on the same preference pairs DPO used (Phase 7's file), with the Bradley-Terry loss: score the chosen response above the rejected one. The result is a dense, smooth reward for any response, a far better gradient signal than the rule's coarse 0 / ⅓ / ⅔ / 1.

Honest caveat, visible at this scale: our reward model only reaches ~61% pairwise accuracy; a 7M linear-head probe on ~1k pairs is a weak judge. RL is only ever as good as its reward model, and ours is the weak link.

RL without the machinery

Second missing piece: a low-variance estimator. RLOO (REINFORCE Leave-One-Out) drops PPO's critic, GAE and clipping entirely. For each prompt it draws k samples and scores them with the reward model; the baseline for each sample is simply the mean of the others. That leave-one-out baseline is the whole variance-reduction trick: no value network, no GAE, no clip. One REINFORCE update per rollout, KL-tethered to the reference.

First attempt, against the ~61% reward model above. RLOO's machinery worked perfectly: the reward-model score tripled (−0.6 → +2.4) and KL to the reference exploded 9× as the policy chased it. But word-inclusion barely budged (22% → 24%). The policy hacked the weak reward model, exploiting its errors instead of using the words. That's reward-model overoptimization (Goodhart's law: "when a measure becomes a target, it ceases to be a good measure"), the central pitfall of RLHF. The reward model, it turned out, didn't even score a 2-word story above a 1-word one, so "maximize it" didn't mean "use the words."

Fix the reward, and it climbs

RL is only as good as its reward model, so we built a better one. The weak RM never learned to grade because its pairs were all-or-none (chosen had every word, rejected had none); it never saw "2 words vs 1." The fix is graded pairs, made for free from the corpus: hold the story fixed and vary the request: ask for words the story happens to contain (3/3 satisfied) vs words it doesn't (1/3). Now the reward model learns to rank by how many requested words appear, and on real generations it does: 2-word stories now score above 1-word above 0.

Same RLOO, same everything; only the reward model changed. And the identical algorithm that hacked now climbs: word-inclusion 22% → 32%, with KL bounded (~4, not 9). It climbed the reward in-distribution, genuinely using more words. The full arc: SFT 22% → PPO 22% → RLOO/weak-RM 24% (hacked) → RLOO/strong-RM 32% (climbed) → DPO 45%. RL from a reward model works, when the reward model is good. And DPO's quiet advantage endures: no reward model to get right in the first place. 🎁 RLOO above is this model. Switch it against 🕹️ PPO on the same words.

Every benchmark has a floor

"45%" invites an obvious complaint: fewer than half the requested words show up, and that's not a good assistant. Fair. But before you accept or dismiss a number, ask what it scores when the model ignores the instruction entirely. TinyStories has a small, repetitive vocabulary, so a story that never read the request still contains ~11% of the requested words by pure coincidence. That's the floor, and it takes one line to measure: score each story against a different prompt's words.

Subtract it and the picture inverts. SFT scores 19.6% against a 10.4% floor: only +9 points of real instruction-following; a third of its headline was luck. DPO scores 43.3% against 11.7%, +32 points. So the honest comparison was never 22% → 45% (a 2× bump); it's 9 → 32, about 3.4×. The raw numbers were understating alignment.

The metric is also harsher than it sounds. The requested words are picked to be distinctive, which means 78% of them are rare. Split it out: on common words DPO hits 60%; on rare words 39%, up from SFT's 13%, nearly 3×. The headline is dominated by the hardest version of the task, and a 7M model with a 4096-token vocabulary may not reliably spell a rare word at all. Some of that miss rate is a capacity ceiling, not an alignment failure.

And one caveat aimed at ourselves: at n=240 the standard error is ±3 points. DPO's 45% vs RLOO's 32% is a real gap (~4 s.e.), but "SFT 22% vs PPO 22%" is just noise agreeing with noise. Read no precision into it. The lesson outlives the number: know your floor, report the lift, and know which part of the task your metric is actually measuring.