Tokenization & Embeddings

Background

Word2Vec (Mikolov et al. 2013) showed word meaning could be encoded as vectors; GPT-2 tokenized subwords with BPE
Tokenization is the critical preprocessing step for all LLMs - the choice of tokenizer affects what the model can learn
Embeddings bridge the discrete world of text to the continuous world of neural network computation

The Concept

Tokenization converts raw text into discrete tokens (subwords, characters, or words). Each token is mapped to a dense vector embedding (learned during training). These embeddings encode semantic meaning: similar meanings cluster together in embedding space.

How it Links

BPE (Byte Pair Encoding) tokenization is used by GPT-family models; SentencePiece by T5 and LLaMA
Embedding layers are the entry point for text into all Transformer models
Retrieval systems like RAG depend on embeddings to find semantically similar documents

Concrete Example

The word "bank" in "river bank" gets a different contextualized embedding than in "bank account" (BERT, contextual embeddings). In word2vec (static embeddings): king - man + woman ≈ queen in vector arithmetic. These vector properties emerge purely from training on co-occurrence patterns.

Prerequisites

Math you need, Multi-Layer Perceptron, RNNs, LSTMs, GRUs, The Transformer Architecture

Learning outcomes

Derive the Word2Vec skip-gram with negative sampling loss from first principles and explain why it can be optimised with a single sigmoid per (word, context) pair.
Compare Word2Vec, GloVe, and FastText on what they factorize and what kinds of out-of-vocabulary words they can handle.
Walk through the BPE merge algorithm step by step on a toy corpus, and explain why byte-level BPE is the modern default.
Compare BPE, WordPiece, Unigram LM, and SentencePiece on training objective and runtime trade-offs.
Diagnose tokenization failure modes: numbers, non-Latin scripts, code, and the SolidGoldMagikarp class of trained-but-untrained tokens.

TL;DR

A language model only ever sees integers. Tokenization turns text into a sequence of integers; embeddings turn each integer into a learned vector. Word2Vec [1][2] introduced trainable distributional embeddings via skip-gram + negative sampling, where each word predicts its context with a single sigmoid per pair. GloVe [3] reframed the same idea as factorizing a global log-co-occurrence matrix. FastText [7] added subword n-grams so out-of-vocab words still get a vector. The other half of the problem is what counts as a token: word-level breaks on rare words, character-level burns sequence length, subword sits in the middle. BPE [4] greedily merges frequent character pairs; WordPiece picks merges that maximize unigram likelihood (used by BERT [9]); Unigram LM [6] prunes from a large initial vocabulary by likelihood; SentencePiece [5] wraps both with a language-agnostic byte- or codepoint-level pre-tokenizer. Modern LLMs use byte-level BPE (GPT-2, Llama, GPT-4 family with tiktoken) so every Unicode string is representable. Contextual embeddings (ELMo [8], BERT [9]) replaced static word vectors entirely; static embeddings now mostly live as the bottom row of a Transformer’s input embedding table.

Intuition

A language model can only look at numbers. The mapping from text to numbers is harder than it looks. Three failure modes drove the entire field:

Word-level vocabularies blow up and miss rare words. English has hundreds of thousands of word forms, plus typos, names, hyphenations. A 50K word vocab leaves rare words and out-of-vocabulary tokens unrepresented; a 500K vocab eats memory and rarely helps.
Character-level sequences are too long. Each English word is on average 4-5 characters, so a 4096-token context window holds about 800 words. Training on character sequences is feasible but pays a $\sim 5$ x compute multiplier [10].
Subword tokenization splits the difference. Common words stay whole (“the”, “running”), rare words split into known pieces (“tokenization” $\to$ “token” + “ization”), and unknown sequences fall back to characters. This is the modern default.

Once you have a token sequence, you need to turn each token into a vector. Two regimes have driven the field. Static embeddings (Word2Vec, GloVe, FastText) give each token one vector regardless of context. Contextual embeddings (ELMo, BERT, every Transformer LM since) give each token a different vector at every position, computed by the model from the surrounding context. Static embeddings are still useful as the bottom row of a Transformer’s input table, and as fast features for retrieval and clustering. Contextual embeddings are what every modern model produces internally.

The distributional hypothesis

“You shall know a word by the company it keeps” (Firth, 1957). All modern embeddings rest on this idea: words that appear in similar contexts mean similar things. Word2Vec operationalizes it by training each word to predict (or be predicted by) its neighbors. GloVe operationalizes it as factorizing the matrix of how often words co-occur. Even contextual embeddings rest on this hypothesis at training time: the BERT masked-LM objective trains a token’s representation by asking it to predict, given context, what word was masked.

The hypothesis fails on context-free polysemy (“bank” = river vs financial), which is exactly where contextual embeddings beat static ones: “river bank” and “savings bank” get different vectors at the same word position.

Static word embeddings

Word2Vec: skip-gram and CBOW

Word2Vec [1] is two related architectures. Skip-gram trains a word to predict its surrounding context; CBOW does the reverse, predicting the center word from the surrounding bag of context words. Skip-gram is more popular for downstream use because it gives slightly better representations for rare words.

Skip-gram with negative sampling: deriving the loss

Naive skip-gram

For a center word $w$ and a context word $c$ in a window of size $C$ , the basic skip-gram model defines

P(c \mid w) = \frac{\exp(u_c^\top v_w)}{\sum_{c' \in V} \exp(u_{c'}^\top v_w)},

where $V$ is the vocabulary, $v_w \in \mathbb{R}^d$ is the input embedding of $w$ , and $u_c \in \mathbb{R}^d$ is the output embedding of $c$ . Each word has two vectors. The training objective is to maximize the log-likelihood of the observed (center, context) pairs:

\mathcal{L}_{\mathrm{naive}} = \sum_{(w, c) \in \mathcal{D}} \log P(c \mid w) = \sum_{(w, c)} \left( u_c^\top v_w - \log \sum_{c'} \exp(u_{c'}^\top v_w) \right).

The gradient of the partition function $\log \sum_{c'} \exp(\cdot)$ is the killer: it is a sum over the entire vocabulary, $|V|$ inner products per training pair. With $|V| = 10^6$ words and $10^9$ training pairs, this is intractable.

Negative sampling

Mikolov et al. [2] replaced the softmax with a binary classification: distinguish real (center, context) pairs from fake pairs sampled at random. For each real pair $(w, c)$ , draw $K$ “negative” words $c_1, \dots, c_K$ from a noise distribution $P_n$ and define the loss

\mathcal{L}_{\mathrm{NEG}}(w, c) = -\log \sigma(u_c^\top v_w) - \sum_{k=1}^K \mathbb{E}_{c_k \sim P_n}\!\left[\log \sigma(-u_{c_k}^\top v_w)\right].

The first term pushes the dot product up for real pairs; the second pushes it down for negatives. Each step costs $O((K + 1) d)$ instead of $O(|V| d)$ . With $K = 5$ to $20$ negatives per positive, training is hundreds of times faster than the full softmax.

Why it works (NCE viewpoint)

Negative sampling is a coarse approximation to noise-contrastive estimation (NCE). NCE poses the question “given a sample, did it come from the data distribution or from a noise distribution?” and shows that the optimal classifier’s logit equals $\log P_d(x) - \log P_n(x)$ up to a constant. So the binary classifier at convergence parameterizes the data distribution, and the embedding $v_w$ that makes this classifier accurate is forced to encode the distributional context of $w$ . The full NCE estimator has consistency guarantees; negative sampling drops the normalization constant, which costs the consistency proof but gains tractability and works well in practice.

Choice of noise distribution

Mikolov chose $P_n(c) \propto P_{\mathrm{unigram}}(c)^{3/4}$ , where the $3/4$ exponent flattens the unigram distribution: rare words become slightly more likely as negatives, frequent words less so. The intuition is that “the” appearing as a negative is uninformative (every word avoids “the” because it appears in every context), while “obscure-rare-word” as a negative is highly informative. The exponent is empirical; values between $0.5$ and $0.75$ all work.

Subsampling frequent words

A second trick: subsample frequent words from the training data with probability $1 - \sqrt{t / f(w)}$ where $f(w)$ is the unigram frequency and $t \approx 10^{-5}$ . This drops most occurrences of “the”, “a”, “of” before they ever enter training pairs. The effect is dramatic: training on the same corpus with subsampling gives better representations for rare words and trains 5-10x faster.

GloVe: global co-occurrence factorization

GloVe [3] recasts word embedding as matrix factorization. Build the word-word co-occurrence count matrix $X$ where $X_{ij}$ is the number of times word $j$ appears in the context of word $i$ (over a sliding window across the corpus). Define the loss

\mathcal{L}_{\mathrm{GloVe}} = \sum_{i, j} f(X_{ij})\, (v_i^\top u_j + b_i + b_j - \log X_{ij})^2,

with a weighting function $f(x) = \min(1, (x / x_{\max})^{3/4})$ that downweights very frequent co-occurrences (the same $3/4$ flattening idea as Word2Vec) and zeros out empty cells. The model factorizes $\log X$ into $V U^\top$ plus per-word biases; each word ends up with two learned vectors, summed at the end to produce the final embedding.

How GloVe and Word2Vec differ in spirit

Both produce static word vectors. The difference is in what they “see”:

Word2Vec is online: it processes one (center, context) pair at a time, never explicitly building the co-occurrence matrix.
GloVe is offline: it builds $X$ once, then optimizes the factorization. Cheaper per step on large corpora because each word pair is processed once, not once per window occurrence.
The two are connected: Levy and Goldberg (2014) showed that Word2Vec with negative sampling implicitly factorizes the shifted PMI matrix $\log(P(w, c) / (P(w) P(c))) - \log K$ . GloVe is more direct about this.

Empirically the embeddings are within noise of each other on most downstream tasks. The choice is mostly about training infrastructure: Word2Vec is a one-pass streaming trainer; GloVe needs the full co-occurrence matrix in memory.

FastText: subword n-grams

FastText [7] replaces the input embedding $v_w$ with a sum over character n-gram embeddings of $w$ . For “where” with $n \in \{3, 4, 5, 6\}$ :

v_{\text{where}} = \sum_{g \in G_{\text{where}}} v_g, \qquad G_{\text{where}} = \{\text{<wh, whe, her, ere, re>, <whe, where, where>}, \dots\}.

(< and > are word-boundary markers.) Two consequences. First, out-of-vocabulary words still get a vector: “tokenization” not in vocab is still composed of “tok”, “oken”, “kenization” subword vectors. Second, morphologically related words share representation: “running”, “ran”, “runs” share enough character n-grams to land near each other in vector space, which is helpful for morphologically rich languages (Finnish, Turkish, Arabic).

The price is vocabulary blow-up: the n-gram set is much larger than the word set. FastText hashes n-grams into a fixed-size bucket array (typically $2 \cdot 10^6$ buckets) to keep memory bounded, accepting hash collisions as a regularization.

Method	Trains on	Loss	OOV handling	Year
Word2Vec skip-gram	(center, context) pairs streamed from corpus	NEG: -log sigma(u_c . v_w) plus K negatives	None (unknown words have no vector)	2013
Word2Vec CBOW	(bag of context, center) pairs	Hierarchical softmax or NEG	None	2013
GloVe	Global word-word log co-occurrence matrix	Weighted least squares on log X_ij	None	2014
FastText	Same skip-gram pairs but with character n-gram inputs	NEG over n-gram-summed embeddings	Yes (compose from known n-grams)	2017
BERT static (input embedding row)	Pretrained masked-LM corpus	Masked-language modelling cross-entropy	Yes (subword fallback at the tokenizer)	2018

Tokenization: subword algorithms

The tokenizer is the layer between raw text and the model. Four algorithms have dominated since 2016.

Byte Pair Encoding (BPE): the merge algorithm

The algorithm

BPE [4] trains a tokenizer by iteratively merging the most frequent pair of symbols in the corpus. The starting alphabet is the set of characters (or bytes); the final vocabulary is characters + a chosen number of merges.

Input: corpus of words pre-split by whitespace (with </w> end-of-word marker)
       target vocab size V

1. Initialize: vocabulary = set of all characters appearing in the corpus.
2. Represent each word as a sequence of characters.
3. Repeat until |vocabulary| = V:
   a. Count how often every adjacent pair of symbols (a, b) appears across the corpus.
   b. Pick the pair with the highest count. Call it (a*, b*).
   c. Add the merged symbol a*b* to the vocabulary.
   d. Replace every occurrence of "a* b*" in the corpus with the single symbol "a*b*".
   e. Record the merge rule "a* + b* -> a*b*" in an ordered list.
4. Output: the vocabulary and the ordered merge list.

At inference, encode a new word by representing it as characters and applying the merge rules in the same order they were learned. Stop when no rule applies. The resulting token sequence is deterministic given the merge list.

A worked example

Corpus (with _ standing in for </w>): low_ low_ low_ low_ low_ lower_ lower_ newest_ newest_ newest_ newest_ newest_ newest_ widest_ widest_ widest_.

Initial vocab: {l, o, w, _, e, r, n, s, t, i, d}.
Most frequent pair across corpus: e s (counts: 6 from “newest_”, 3 from “widest_” = 9). Merge to es. Vocab becomes {..., es}.
Next: es t (9). Merge to est.
Next: est _ (9). Merge to est_.
Next: l o (5+2=7). Merge to lo.
Next: lo w (7). Merge to low.
…

After 10 merges, “lower_” tokenizes as low e r _. After 20 merges with the right schedule, it tokenizes as low er_.

Byte-level BPE (BBPE)

The original BPE in [4] ran on Unicode characters. GPT-2 introduced byte-level BPE: run BPE on the raw byte sequence (UTF-8 encoded). Two consequences:

Universal coverage. Every Unicode string is a sequence of bytes, so the tokenizer never produces an <unk> token. Emoji, ancient scripts, broken UTF-8, all encode to bytes and tokenize to known merges.
Roughly 256-base alphabet. The starting vocab is just the 256 bytes (often pre-mapped to printable characters to keep the trained tokenizer human-readable).

GPT-2, GPT-3, GPT-4 (with tiktoken), Llama, Mistral, and most modern LLMs use byte-level BPE. The single notable exception is BERT, which used WordPiece on Unicode characters (with [UNK] for missing chars) in 2018; BERT-derived models still inherit that.

Why merge order matters

BPE applies merges in the order they were learned, so changing the order changes the segmentation. This is a feature (deterministic), not a bug. It does mean that if you train a BPE on English and apply it to French, the merges will be biased toward English digrams; the French word “fenetre” might tokenize into many small pieces, costing context length. Production multilingual tokenizers (mT5, XLM-R) train on a balanced multilingual corpus to avoid this.

WordPiece, Unigram LM, and SentencePiece

Three siblings to BPE differ in how they pick what to keep in the vocabulary.

WordPiece (used by BERT [9]). Like BPE, but instead of picking the most frequent pair, it picks the pair $(a, b)$ that maximizes the likelihood gain under a unigram language model: roughly, the pair $(a, b)$ for which $\log P(ab) - \log P(a) - \log P(b)$ is largest. This biases toward pairs that occur together more than chance, not just often. WordPiece-trained vocabs put a ## prefix on continuation pieces (##ization is a continuation; token is a word start).
Unigram LM [6]. Start with a large initial vocabulary (BPE-trained, or all substrings up to length $k$ ), define a unigram language model over the vocabulary, and prune the lowest-likelihood pieces iteratively until the target size is hit. At inference, the segmentation is the most likely sequence under the unigram LM (computed by Viterbi over a forward DAG). Used by ALBERT, T5 (via SentencePiece), and Gemma.
SentencePiece [5]. Not a tokenization algorithm but a language-agnostic toolkit that wraps either BPE or Unigram LM and runs on raw text without language-specific pre-tokenization. The key engineering choice: SentencePiece treats whitespace as a regular character (encoded as _ U+2581) so detokenization is lossless. T5, ALBERT, XLNet, and most non-English models use SentencePiece.

Algorithm	Training objective	Direction	Inference cost	OOV	Notable users
BPE	Greedy: merge most frequent pair	Bottom-up (chars to subwords)	Apply ordered merges (linear)	No (chars/bytes always available)	GPT-2/3/4, Llama, Mistral, RoBERTa
Byte-level BPE	Greedy: merge most frequent byte pair	Bottom-up (bytes to subwords)	Apply ordered merges	Never (every byte string covered)	GPT-2/3/4 (tiktoken), Llama, Mistral
WordPiece	Greedy: merge highest likelihood gain (log P(ab) - log P(a) P(b))	Bottom-up	Greedy longest-match	Yes (rare; falls back to [UNK])	BERT, DistilBERT, ELECTRA
Unigram LM	Prune low-prob pieces from large initial vocab	Top-down	Viterbi over forward DAG	Falls back to single chars (UNK)	T5 (with SentencePiece), Gemma, ALBERT
SentencePiece (toolkit)	Wraps BPE or Unigram with language-agnostic pre-tokenization	Either direction	Same as wrapped algo	Falls back to chars/bytes	T5, mT5, XLNet, Gemma, ALBERT

Vocabulary size trade-offs

Vocab size	Sequence length on English text	Embedding-table cost (d=2048)	OOV behavior	Typical users
256 (byte-level)	~5x word count	~ 0.5 MB	None (every byte mapped)	ByT5, character-level transformers
8K	~2x word count	~ 16 MB	Subwords always available	Small specialized LMs
32K-50K	~1.3x word count	~ 100 MB	Subwords cover almost everything	BERT, GPT-2, Llama-2, Mistral
100K-128K	~1.1x word count	~ 250 MB	Common multilingual or code-trained	Llama-3 (128K), GPT-4 (~100K via tiktoken)
250K+	~1.05x word count	~ 500 MB	Full multilingual with rare languages	mT5, XLM-R, NLLB

The trade-off in one line: bigger vocabulary = shorter sequences (less compute per training step) and a larger input/output embedding table (more parameters and memory). Modern LLMs trend toward larger vocabularies (Llama-3 at 128K is 4x Llama-2’s 32K) because longer context windows and multilingual capability favor the bigger vocab.

Contextual embeddings

Static embeddings give one vector per word. Contextual embeddings give a different vector for each occurrence based on surrounding context.

ELMo [8] uses a deep bidirectional LSTM trained as a language model. The contextual representation of a token is a learned linear combination of the bidirectional LSTM’s hidden states across layers. Pre-Transformer SOTA on most NLP tasks in 2018.
BERT [9] replaces the LSTM with a Transformer encoder and trains it with masked language modelling: 15% of tokens are randomly masked and the model predicts the masked tokens from context. The bidirectional Transformer pre-training, plus the simplicity of fine-tuning, made BERT the dominant pre-trained encoder in 2018-2020.

A key practical fact: the input embedding row of every modern Transformer is a static embedding. You can extract it as a $V \times d$ matrix and use it like a Word2Vec vector. It will be slightly worse than dedicated static training because the model “spends” representation capacity on context-aware features in upper layers, but it is free if you already have the model. Cross-link: Encoder, Decoder & Encoder-Decoder Models covers BERT vs GPT vs T5 in depth.

The tokenizer eats compute

A common mistake: assuming that swapping a 32K English-trained BPE for a 128K multilingual one is “free” because both produce token sequences. In reality, the multilingual one usually produces shorter sequences for English text (because more frequent English n-grams got merged into single tokens), but the embedding table is 4x larger and the output projection 4x more expensive. For a 7B model, the vocab embedding alone goes from $\sim 100$ M to $\sim 400$ M parameters; at 128K vocab, the final softmax becomes a bandwidth-significant slice of decode latency.

The other side: a 32K vocab on Hindi or Tamil text might tokenize each word into 5-8 pieces (every char becomes its own token), inflating sequence length 5-8x. Compute per training step is dominated by sequence-length-times-vocab-size at the output, so a “bad” tokenizer for the language you actually use is a real compute tax. This is why every serious multilingual model retrains its tokenizer.

Interactive: BPE tokenizer

The visual demo first. Step through 15 precomputed BPE merges on the default sentence “the quick brown fox jumped over the lazy dog”. Each colored box is a token; words the merge table never covers (jumped, over, lazy) stay split into more pieces, exactly what a real subword tokenizer does on out-of-distribution input.

BPE tokenizer (precomputed, 15 merges)

Input text (lowercase a-z and spaces only):

Step 0 / 15

the_

quick_

brown_

fox_

jumped_

over_

the_

lazy_

dog_

Show merge order (15 merges)

t + h → th
th + e_ → the_
o + w → ow
q + u → qu
b + r → br
br + o → bro
bro + w → brow
brow + n_ → brown_
qu + i → qui
qui + c → quic
quic + k_ → quick_
f + o → fo
fo + x_ → fox_
d + o → do
do + g_ → dog_

Each colored box is one token. The _ suffix marks the end of a word (the </w> symbol from the BPE paper). Words the merge table never covers, like "jumped" or "over", stay split into more pieces, which is exactly what a real subword vocabulary does on out-of-distribution input.

Implementations

Three idiomatic implementations: HuggingFace tokenizers for byte-level BPE, the sentencepiece Python library for Unigram LM, and gensim for Word2Vec. Pick the one that matches your stack.

python

# Train a byte-level BPE tokenizer on a small corpus, then encode and decode.
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel

tokenizer = Tokenizer(BPE(unk_token=None))
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=False)

trainer = BpeTrainer(
  vocab_size=4096,
  initial_alphabet=ByteLevel.alphabet(),  # all 256 byte mappings
  special_tokens=["<pad>", "<bos>", "<eos>"],
)

# Train on a small text file (one sentence per line, in practice ~1 GB).
tokenizer.train(["wikitext-103-tiny.txt"], trainer)
tokenizer.save("byte-bpe.json")

enc = tokenizer.encode("Tokenization eats compute.")
print(enc.tokens)   # e.g. ['Token', 'ization', ' eats', ' compute', '.']
print(enc.ids)      # the integer ids

# Round-trip:
print(tokenizer.decode(enc.ids))   # 'Tokenization eats compute.'

Byte-level BPE never emits <unk>: every Unicode string is a sequence of bytes that maps to known merges. The initial_alphabet=ByteLevel.alphabet() seeds the vocabulary with all 256 bytes so even unseen characters tokenize cleanly.

Run a tiny in-browser version of skip-gram negative sampling on a toy corpus:

Idle (packages: numpy)

import numpy as np

np.random.seed(0)

corpus = ['the quick brown fox jumps over the lazy dog',
        'a quick brown fox jumps over a lazy cat',
        'the lazy dog sleeps under the warm sun']
tokens = [w for s in corpus for w in s.split()]
vocab = sorted(set(tokens))
w2i = {w: i for i, w in enumerate(vocab)}

V, d = len(vocab), 16
v_in = np.random.randn(V, d) * 0.1   # input embeddings
v_out = np.random.randn(V, d) * 0.1  # output embeddings

# Build (center, context) pairs with window 2.
pairs = []
window = 2
sents = [s.split() for s in corpus]
for s in sents:
  for i, w in enumerate(s):
      for j in range(max(0, i - window), min(len(s), i + window + 1)):
          if j != i:
              pairs.append((w2i[w], w2i[s[j]]))

# Skip-gram with negative sampling, K=5 negatives.
def sigmoid(z): return 1 / (1 + np.exp(-z))
unigram = np.bincount([w2i[w] for w in tokens], minlength=V).astype(float)
neg_p = (unigram ** 0.75); neg_p /= neg_p.sum()
K, lr = 5, 0.05

for epoch in range(50):
  np.random.shuffle(pairs)
  loss = 0.0
  for w, c in pairs:
      score_pos = v_in[w] @ v_out[c]
      loss += -np.log(sigmoid(score_pos) + 1e-12)
      grad = (sigmoid(score_pos) - 1)
      v_out[c] -= lr * grad * v_in[w]
      v_in[w]  -= lr * grad * v_out[c]
      # K negatives per positive
      for _ in range(K):
          n = int(np.random.choice(V, p=neg_p))
          score_neg = v_in[w] @ v_out[n]
          loss += -np.log(sigmoid(-score_neg) + 1e-12)
          grad_n = sigmoid(score_neg)
          v_out[n] -= lr * grad_n * v_in[w]
          v_in[w]  -= lr * grad_n * v_out[n]
  if epoch % 10 == 0:
      print(f'epoch {epoch:2d}: loss {loss / len(pairs):.4f}')

# Cosine similarity probe.
def cos(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print()
print('cos(fox, dog):', round(cos(v_in[w2i['fox']], v_in[w2i['dog']]), 3))
print('cos(fox, the):', round(cos(v_in[w2i['fox']], v_in[w2i['the']]), 3))

(no output yet)

Pitfalls

Numbers and tokens

Most BPE tokenizers split numbers in surprising ways. 123456 might tokenize as 12 34 56 or 1234 56 depending on which digit n-grams the trainer happened to merge. This makes arithmetic and number reasoning hard for LLMs: the model sees 123 + 456 = ? not as three numbers but as some token-soup whose pieces depend on the merge order. Modern fixes: explicitly split digits one-by-one in the pre-tokenizer (Llama-3, GPT-4 with tiktoken’s o200k_base), or train a tokenizer with a \d+ regex that forces digit boundaries.

Non-Latin scripts and the multilingual tax

A 32K BPE trained on English will tokenize Hindi or Korean character by character (or worse, byte by byte for byte-level BPE), inflating sequence length 5-10x. The model sees a much smaller fraction of the actual content per context window. Production multilingual models (mT5, XLM-R, BLOOM, Llama-3) retrain on a balanced multilingual corpus, often at 128K-250K vocab. If you fine-tune an English-trained model on a non-Latin language and quality is poor, the tokenizer is usually the bottleneck before the model is.

Code is not text

BPE trained on natural language tokenizes code badly: indentation becomes opaque, identifiers like getUserById split inconsistently, and language keywords are not whole tokens. Code-specific models (CodeLlama, StarCoder, GPT-4 Code Interpreter) retrain the tokenizer on a code corpus, giving keyword-level merges (def, function, return) and a treatment of indentation. If you fine-tune a general LLM on code without retraining the tokenizer, sequence lengths blow up and the model spends capacity on lexing.

The SolidGoldMagikarp class of trained-but-untrained tokens

A famous failure mode of GPT-2 and GPT-3: certain rare tokens (SolidGoldMagikarp, davidjl, ?\?\?\?\?-?\?\?\?\?-) appear in the BPE vocabulary because they showed up frequently in the tokenizer training corpus (Reddit usernames, scraped artifacts) but not in the language-model training corpus that came later. The model has an embedding for these tokens but never updated it during pretraining. Asking the model about SolidGoldMagikarp triggers wildly inconsistent behavior: refusal, hallucination, or word-salad responses. The fix is structural: train the tokenizer and the LM on the same data, and audit the rarest tokens before locking the vocab.

Always inspect the tokenization of your eval prompts

Before assuming a model is bad at a task, paste the prompt into the model’s tokenizer and read the token list. “When was Albert Einstein born?” might tokenize differently in two models, and a model that tokenizes “Einstein” as Ein stein cannot leverage all the “Einstein”-specific knowledge it has from pretraining. This is a 30-second sanity check that catches more “model failures” than people realize.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. In Word2Vec skip-gram with negative sampling, the loss for one positive (center w, context c) plus K negative samples is:
2. Why is byte-level BPE the default for multilingual LLMs?
3. GloVe optimizes a weighted least-squares loss against:
4. WordPiece picks merges differently from BPE because:
5. Why does FastText handle out-of-vocabulary words better than Word2Vec?

Interview Q&A

Derive the Word2Vec skip-gram negative sampling loss from first principles. Why is it tractable when the full softmax is not?onsite

Naive skip-gram. For a center word $w$ and a context word $c$ in a window, the model defines

P(c \mid w) = \frac{\exp(u_c^\top v_w)}{\sum_{c' \in V} \exp(u_{c'}^\top v_w)},

with input embeddings $v_w$ and output embeddings $u_c$ . Training maximizes $\sum_{(w, c)} \log P(c \mid w)$ . The gradient of the log-partition $\log \sum_{c'} \exp(u_{c'}^\top v_w)$ requires summing over all $|V|$ words, which costs $O(|V| d)$ per training pair. With $|V| = 10^6$ and $10^9$ pairs, this is $10^{15}$ inner products, intractable on 2013 hardware.

Negative sampling. Replace the multiclass softmax with a binary classifier: distinguish real $(w, c)$ pairs (label 1) from fake $(w, c_k)$ pairs (label 0) where $c_k \sim P_n$ is drawn from a noise distribution. For each real pair $(w, c)$ and $K$ negatives $c_1, \dots, c_K$ , the loss is

\mathcal{L}_{\mathrm{NEG}}(w, c) = -\log \sigma(u_c^\top v_w) - \sum_{k=1}^K \log \sigma(-u_{c_k}^\top v_w).

The first term pushes the dot product up for real pairs (sigmoid toward 1); the second pushes it down for negatives (sigmoid toward 0). Each step costs $O((K + 1) d)$ .

Why it’s correct (NCE viewpoint). Negative sampling is a coarse approximation to noise-contrastive estimation. NCE trains a classifier to distinguish data from noise; under mild conditions, the optimal classifier’s logit is $\log P_d(x) - \log P_n(x)$ . The classifier’s parameters at convergence implicitly parameterize the data distribution. Negative sampling drops the constant log-noise term, losing NCE’s consistency proof but keeping the practical objective: pull $v_w$ toward contexts it actually appears with, push it away from random other words.

Why it’s fast. The full softmax couples every output unit through the partition function. Each gradient step requires re-evaluating the partition. Negative sampling decouples: each (positive, negatives) batch is a fixed-size, independent computation. With $K = 5$ negatives and $d = 300$ , one update is $\sim 1800$ operations, vs $300 \cdot 10^6 = 3 \cdot 10^8$ for the full softmax. The 100,000x speedup is what made Word2Vec practical at billion-word scale [2].

Choice of noise. Mikolov’s empirical recipe: $P_n(c) \propto P_{\mathrm{unigram}}(c)^{3/4}$ . The 3/4 exponent flattens the distribution: rare words become slightly likelier as negatives, frequent words less so. The intuition is that “the” as a negative is uninformative (every word avoids “the”), while a rare word as a negative carries real information.

Why is byte-level BPE the modern default for multilingual LLMs? Compare with WordPiece and Unigram LM.onsite

Byte-level BPE wins on coverage. The tokenizer’s job is to map every input string to a sequence of integers. Byte-level BPE starts from the 256 bytes (UTF-8 encoding) and merges from there. Every Unicode string is a sequence of bytes, so the tokenizer never returns <unk>: emoji, ancient scripts, broken UTF-8, code, all encode to bytes and tokenize to known merges. This matters at deployment: a chatbot facing real users sees Polish, Tamil, mixed code, emoji, and broken inputs. A tokenizer that emits <unk> is a model that can’t even read the prompt.

WordPiece (BERT) starts from Unicode characters. If a character is missing from the trainer’s seen set (rare scripts, emoji), it falls back to [UNK]. This was acceptable for BERT (English-only, news/Wikipedia corpus) but is a real limitation for any multilingual or user-facing system. Byte-level BPE eliminates the failure mode.

Unigram LM (T5 via SentencePiece) trains a probabilistic model over the vocabulary and segments via Viterbi. Pros: probabilistic segmentation enables subword regularization (sample alternative segmentations during training, which is regularization for free). Cons: more expensive at training time (need to estimate the unigram LM iteratively); tied to whatever character or byte alphabet you start with. T5 uses SentencePiece + Unigram LM with character coverage 99.95% and falls back to bytes for the remaining 0.05%, which is a hybrid that captures most of the byte-level BPE advantage with the regularization win.

Trade-off matrix. For a brand-new multilingual LLM trained from scratch in 2024:

Byte-level BPE: simple, universal coverage, no <unk>, no special handling needed for any language. The default. Used by GPT-2/3/4 (tiktoken), Llama, Mistral, RoBERTa.
SentencePiece + Unigram LM: better regularization, slightly more complex training. Used by T5, mT5, ALBERT, Gemma.
WordPiece: legacy. Still used by BERT-derived models for compatibility.

The remaining open question is digit handling. Byte-level BPE merges digits like any other bytes, producing surprising tokenizations of numbers (“123456” can become any number of pieces). Llama-3 and GPT-4’s o200k_base force per-digit splits in the pre-tokenizer regex, getting better arithmetic at the cost of slightly longer sequences for numerical text.

What breaks tokenization, in production? Walk through three real failure modes.onsite

1. Numbers and arithmetic. A standard BPE tokenizer trained on natural language merges digit n-grams unpredictably: “1234567890” might tokenize as 12 345 67 890 or 1 234 5678 90 depending on the merge order learned during BPE training. The LM then sees arithmetic problems as token-soup whose pieces depend on training-set artifacts, not as numbers. Llama-3 and GPT-4 force per-digit splits at the pre-tokenizer; this is one of the simplest tokenizer changes that meaningfully improves arithmetic accuracy.

2. Multilingual vocabulary tax. A tokenizer trained predominantly on English will tokenize Hindi or Tamil at near-character granularity. A 5-word Hindi sentence becomes 30+ tokens, blowing past context length and forcing the model to spend capacity on character-level lexing instead of meaning. The fix is structural: balance the multilingual training corpus (mT5, XLM-R, BLOOM, Llama-3 do this) so the BPE learns frequent n-grams in every target language. Inflation factor under a poor tokenizer can be 5-10x for non-Latin scripts; under a good multilingual one, it drops to about 1.5x.

3. Trained-but-untrained tokens (the SolidGoldMagikarp class). A tokenizer trained on a different corpus from the language model produces tokens that the LM has never seen any examples of. These tokens have an embedding (the random init from the start of LM training) but no signal to update it. Asking the model about such tokens triggers undefined behavior: refusals, hallucinations, word-salad responses. The original SolidGoldMagikarp examples come from GPT-2’s tokenizer being trained on Reddit (where these usernames appear) while the LM was trained on a curated subset that excluded them.

Mitigations:

Train the tokenizer and the LM on the same data.
Audit the rarest tokens in the vocabulary before locking it. Tokens with fewer than $\sim 1000$ occurrences in the LM training corpus are suspect.
Have the LM training pipeline emit a histogram of per-token loss; tokens with anomalously high loss either need more data or should be dropped from the vocab.
For inference safety: red-team the tokenizer with fuzzed inputs (Unicode normalization edge cases, zero-width joiners, byte-order marks) to confirm round-tripping.

The general principle: the tokenizer is part of the model, not infrastructure. It must be co-designed with the training data and with the deployment surface. A “good” tokenizer for English news is a “bad” tokenizer for code, and a “good” tokenizer for chat is a “bad” tokenizer for math.

The SolidGoldMagikarp anomaly: what is it, why does it happen, and how would you prevent it in a new model?research

The anomaly. GPT-2 and GPT-3 contain BPE tokens like SolidGoldMagikarp, davidjl, \?\?\?-\?\?\?\?-?, and a handful of others that trigger bizarre behavior when used in prompts: refusals, hallucinations, repetition loops, or completely off-topic responses. The behavior is reproducible and consistent across temperatures. At first sight it looks like a model bug; it is actually a tokenizer bug with downstream effects.

Root cause. GPT-2’s BPE tokenizer was trained on a corpus that included a lot of Reddit content, where these strings appeared as usernames or repeated artifacts of scraping. The BPE learned them as single tokens because they were frequent in the tokenizer training corpus. GPT-2’s language-model training corpus, however, was a different, smaller, curated subset that excluded most of the Reddit content. So the LM has an embedding row for SolidGoldMagikarp (created during tokenizer training and inherited by the LM’s vocab), but never saw any examples of that token during LM training. The embedding is whatever random initialization gave it on day 1, frozen because no gradient ever flowed through.

Why behavior is undefined. The model’s first layer maps SolidGoldMagikarp to its random initial vector. From the model’s perspective, that vector is uncorrelated with any training distribution: it activates whatever neurons happen to fire on random Gaussian noise. Whatever those neurons do downstream is the model’s “interpretation” of the token. Different rare untrained tokens activate different random combinations, producing inconsistent and bizarre outputs.

Prevention for a new model.

Train the tokenizer on the same corpus as the language model. This is the single most important fix. If a token is in the vocabulary, the LM has seen it.
Audit the rarest tokens before locking the vocabulary. Sort tokens by frequency in the LM training corpus; flag any token below some threshold (1000 occurrences is a reasonable floor) and decide whether to drop it from the vocab or augment the corpus.
Monitor per-token training loss and end-of-training loss. A token whose final loss is much higher than the average is likely undertrained; investigate and either retrain with more data containing that token, or remap it.
Pretokenizer regexes that prevent absurd token formations. GPT-4’s o200k_base uses a more aggressive pretokenizer regex that prevents long alphanumeric sequences from being merged into a single token, which structurally rules out the SolidGoldMagikarp class.
Vocabulary refresh between iterations. When fine-tuning on a new domain, audit the tokenizer for the new domain’s vocabulary; consider extending the vocab with domain-specific tokens (and initializing them as the average of their constituent subwords).

The bug is illustrative of a broader principle: tokenizer training and LM training must share their data distribution. Any mismatch produces tokens whose embeddings are not grounded in any usage. The same logic applies, less dramatically, to fine-tuning: a token that appears 50,000 times in pretraining and 0 times in fine-tuning will keep its pretraining-era embedding through fine-tuning, which is what you want; a token that is new to fine-tuning (because the tokenizer was extended) needs deliberate initialization.

Build it yourself

A weekend project that takes tokenization from formula to working pipeline:

Implement BPE training from scratch in NumPy on the WikiText-103-tiny dataset. Train to a 4096-vocab tokenizer; verify it matches the HuggingFace tokenizers BPE on the first 100 merges.
Train a Word2Vec skip-gram with negative sampling on the same corpus using the in-browser PyRunner above (or a faster numpy version offline). Compute cos(king - man + woman, queen) and verify it’s near 1.
Swap to byte-level BPE and observe how the merge list differs (more merges spent on common byte n-grams, fewer on rare characters). Verify that the tokenizer round-trips arbitrary UTF-8 strings (emoji, Hindi, code).
Stretch: train the same vocab with SentencePiece + Unigram LM, then feed both tokenizations through the same downstream classifier (sentiment, NER) and compare.
Stretch: train a tiny BERT-style encoder on this corpus with masked-LM and confirm that the input embedding rows are usable as static embeddings (compare cosine similarities to the gensim Word2Vec ones).

References

Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR Workshop 2013 (arXiv preprint). arXiv:1301.3781
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. NeurIPS 2013. arXiv:1310.4546
Pennington, J., Socher, R., and Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. EMNLP 2014. https://nlp.stanford.edu/pubs/glove.pdf
Sennrich, R., Haddow, B., and Birch, A. (2015). Neural Machine Translation of Rare Words with Subword Units. ACL 2016 (arXiv preprint 2015). arXiv:1508.07909
Kudo, T. and Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018 demo. arXiv:1808.06226
Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018. arXiv:1804.10959
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching Word Vectors with Subword Information. TACL 2017. arXiv:1607.04606
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. NAACL 2018. arXiv:1802.05365
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019 (arXiv preprint 2018). arXiv:1810.04805
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

Transformers Encoder, Decoder & Encoder-Decoder Models