The Words GPT Taught Itself

Two strings. “9999”, and “ SolidGoldMagikarp” — a Reddit username, complete with the space in front of it, the way a word sits in a sentence. To GPT-2, each is a single word in its vocabulary. And “1776”, the most famous year in American history, is not: it arrives in two pieces. Nobody at OpenAI ruled that a Pokémon username deserved to be a word while the Declaration of Independence didn’t. A counting loop made those calls, about fifty thousand of them.

recomputed · tiktoken 0.13.0 · r50k_base

The loop is older than the models it feeds. It was invented to shrink files on a 486 and hired twice more before it met the internet.

Gage’s 1994 version fits in a sentence: find the most frequent pair of adjacent bytes, replace every copy with a byte the file never uses, repeat until nothing repeats.

GPT-2’s turn made the “byte” literal again: its alphabet is the 256 possible byte values, so any string you can type is representable, and “unknown token” is impossible by construction. What stayed hand-written is one regex that pre-chops text before any merging happens. The regex is where policy lives.

One costume to see through before the later plates. Token dumps from this era are full of Ġ, and people treat it like a mystery rune. It’s byte 32, the space character, dressed up by the serialization format. It opens most word tokens, because in token-land the space in front is part of the word:

That is the whole machine: count every adjacent pair, merge the winner into a new token, repeat — about 50,000 times for GPT-2, whose 50,257 slots are 256 bytes + 50,000 learned merges + 1 special token. Frequency proposes, the regex vetoes, and nothing in the loop knows what any of it means.

Reading about the algorithm is one thing. Watching it run is better, so we trained one: about 170 lines of Python, fed Pride and Prejudice, Moby-Dick, 1.3 MB of CPython standard-library source, and 400 KB of generated numbers and dates — roughly 3.6 MB in all, for 1,000 merges. Every number below is read back out of the run’s own logs.

Merge number one: two spaces become one token. Merge two: four spaces. Before it learns “the”, before any English at all, it learns code indentation. In a corpus that is roughly a third Python source, whitespace is the most repeated thing there is. The first whole word arrives at merge 9: the, leading space included.

and lands at #46. At #126 comes self — mostly the Python’s doing, with an assist from prose words like “selfish”. By #682 Ahab is a single token; Elizabeth follows at #690. Nobody told the counter these were names; Melville and Austen just repeated them often enough.

The cruel part is the cutoff. Queequeg appears 253 times in Moby-Dick — a major character, and not enough. He misses the thousand-merge cut. Extend the run to 1,500 and the bare Queequeg fuses at #1054; the leading-space form Ahab and Elizabeth got trails in at #1232. A vocabulary is a ranked competition with a hard cutoff, and just below the line, you stay in pieces.

Steep gains early, smaller late, as the best patterns get used up. Scale the merge count up fifty-fold and you have the real thing.

The real thing is called r50k_base: 50,257 tokens, learned around 2019, used by GPT-2 and then GPT-3. OpenAI never published what text it was trained on. But a BPE vocabulary is that text’s frequency table, frozen, and we can read it the way a fossil record gets read.

Start with the numbers, because the numbers are where it gets personal. Every year from 1900 to 1999, with a leading space, the way a year sits in a sentence, is a single token. All one hundred of them.

The same logic decides which numbers exist at all. r50k holds 907 bare single-token integers below 10,000, whichever ones the internet repeated. Compare three of them:

Zoom out and the whole vocabulary tells one story. Run a crude classifier over all 50,256 mergeable tokens and about 93% land in buckets classified as English words and word fragments. The Han-script bucket, covering characters Chinese and Japanese share, holds 42 tokens. Not 42 percent. Forty-two tokens.

Scroll to the longest tokens and you find the internet’s sediment. Separator walls off forum posts and README files. And a ladder of mojibake: text decoded wrong, re-encoded, re-posted, and re-broken so many times that the corruption itself earned vocabulary slots, in exact powers of two.

And filed between two perfectly ordinary entries, at slot 43,453, sits one username, space and all.

Early 2023. Jessica Rumbelow and Matthew Watkins were running k-means over the token embeddings of GPT-J — an open model that shares GPT-2's 50,257-token vocabulary. As a probe, they looked at the tokens nearest the centroid of the whole embedding cloud, where you'd expect the blandest, most average entries in the dictionary. What they found was junk: attRot (id 35207), EStreamFrame (43177), SolidGoldMagikarp (43453).

So they did the simplest thing you can do with a language model: asked OpenAI's models to repeat the strings back. The models couldn't.

These prompts weren't adversarial; the words were simply unspeakable. The archaeology came next, and most of the weird tokens traced to specific, heavily repeated corners of the pre-2019 web.

Why would a model choke on entries in its own dictionary? Here is the careful version.

The first half of the story — how a username becomes a token — reenacts cleanly. We planted the string SolidGoldMagikarp into the toy corpus from Plate II 2,000 times, about one percent of its bytes, and retrained.

Getting into the dictionary is that easy: frequency is the only entry fee. Being understood by a model is a separate transaction — and the hypothesis is that the glitch tokens never completed it.

OpenAI fixed its tokenizer in generations, and each fix is its own lesson. 2021: the Codex tokenizer, built for code. Diff p50k_base against GPT-2’s r50k and the entire upgrade is 24 appended tokens: runs of two to twenty-five spaces, nothing else changed. Sixteen spaces of indentation had cost sixteen tokens. Now they cost one.

2022: cl100k_base, the ChatGPT and GPT-4 tokenizer. Twice the ID space (100,277 slots against 50,257) plus new hand-written rules; whether the vocabulary was retrained from scratch, OpenAI has never said. The smallest new rule answers a four-year-old apology sitting in GPT-2’s source:

cl100k’s regex opens with '(?i:[sdmt]|ll|ve|re). One line, apology accepted. The bigger rule change hit numbers: digit runs now cap at three, so no four-digit token like 9999 can exist at all.

Was arithmetic saved? Not quite, and the reason is sneaky. cl100k chunks digits left to right, but arithmetic carries right to left, ones-tens-hundreds. Write the same numbers with commas and the chunks snap to place value:

Singh & Strouse measured GPT-3.5 and GPT-4 addition accuracy “up to 20% higher” with that comma-forced alignment. The model untouched; only the token boundaries moved.

LaTeX never got a patch at all: on the two arXiv papers we measured, math markup cost about 1.3× prose tokens per character, and that ratio barely moved across three vocabulary generations (1.36→1.29).

And the haunted tokens? Most of Plate IV’s specimens shatter into ordinary pieces under cl100k — but 45 of the 141 candidate strings survive as single tokens. In June 2023, users found GPT-4 still couldn’t say davidjl (cl100k id 99944), the token behind r/counting’s #1 counter, roughly 163,000 counting posts. Its documented substitutions: “jndl”, “jspb”, “JDL”. And cl100k had minted code-shaped undertrained tokens of its own:

2024: o200k_base, the GPT-4o tokenizer, 200,000 slots and multilingual at last. Entries classified as Han script jump from 42 to 7,397 (our first-match regex heuristic), ␣davidjl finally splits in two, and every language premium we measured falls:

Then researchers sorted o200k’s longest Chinese entries. The list reads like a spam folder:

The tokens themselves are the receipts: walls of porn and gambling spam got counted somewhere in the pipeline that built this vocabulary, in its training text or its token selection. OpenAI has never disclosed either. Within weeks, researchers were using tokens like these to make GPT-4o hallucinate and, occasionally, slip past its guardrails. A new language, a bigger model, the same kind of scar.

Every plate so far read the fossil record. This one re-runs the deposition. In July 2026 we streamed 2.00 GiB of OpenWebText — the public recreation of GPT-2’s WebText training corpus — into Hugging Face’s byte-level BPE trainer, with GPT-2’s split regex, the same 50,257-token budget, and min_frequency=2. Different implementation, different decade, roughly 5% of OpenAI’s data volume. How much of GPT-2’s dictionary comes back?

The pattern is clean. The head of the vocabulary — common words, years, formatting — is corpus-robust: any big-enough sample of the same distribution rediscovers it almost token for token, and anyone with a laptop can re-derive it in about a minute. The scars are corpus-specific. OpenWebText is deduplicated and cleaned, so the double-encoded mojibake and the r/counting ghosts, which lived in the dirty corners of the original crawl, never re-form. Instead the sample mints its own junk. That sharpens this dossier’s thesis: a tokenizer is a fossil record, and it is precisely the weird tokens that identify the stratum.

Every plate above comes down to the same move: hand a string to four vocabularies and watch where the knife falls. This drawer holds 64 specimens, pinned. Each card shows one string as r50k (GPT-2, GPT-3), p50k (Codex), cl100k (ChatGPT, GPT-4), and o200k (GPT-4o) cut it, with the token count at the end of each row.

A ␣ marks a leading space, and it belongs to the token — the and the are different vocabulary entries. Dashed chips are byte fragments that aren't valid UTF-8 on their own (each dot is one raw byte); byte-level BPE is allowed to cut mid-character, and for emoji and some scripts it does.

Worth trying: search 1999 for a year-wall resident, then 1776 to see the same era split 17|76 in r50k and 177|6 in cl100k. Search davidjl for the famous survivor. Toggle glitch and spam to browse the fossil bed; toggle whitespace to watch sixteen tokens of indentation become one.