Tokenization and Embeddings: How Text Becomes Numbers (BPE by Hand)

Here is a strange fact. A language model has never read a single word. It has never seen a letter, a space, or a comma. All it ever receives is a list of integers, and everything it knows about language is built on top of that list. Today we learn exactly how text becomes those integers, and then how those integers become meaningful vectors.

• Models only see integers
• Text must be converted first

In lesson 53 we studied LSTMs, GRUs, and backpropagation through time, which handle sequences one step at a time. Before we can build transformers, we need to settle the input side. Today covers tokenization, embeddings, and positional information. Attention itself is not in this lesson. That is lesson 55, and everything today prepares you for it.

• Lesson 53: LSTM, GRU, BPTT
• Lesson 54: tokens and embeddings
• Attention comes next

A neural network multiplies, adds, and differentiates. As we saw in lesson 26, every layer is a matrix operation, and learning needs gradients. You cannot take the derivative of the letter c. So we need a fixed, reversible mapping from pieces of text to numbers, and later a way to turn those numbers into vectors the network can actually learn from.

• Multiply, add, differentiate
• Letters have no gradient

Suppose we give every word an id: cat is five, dog is six, and carburetor is seven. Feeding those directly would suggest dog is bigger than cat, and that seven is one step beyond six. The ids are only names, like jersey numbers. So integers are used as addresses to look things up, not as quantities. Keep that idea, because the embedding matrix is built on it.

• IDs are labels, not quantities
• cat=5, dog=6 means nothing

The first idea is one token per word. Split on spaces and give each distinct word an id. Sentences become short, which is nice. But English alone has hundreds of thousands of word forms, and run, runs, running, and runner each get separate ids. Worse, any word not in the table, like a new product name or a typo, has nowhere to go.

• One token per word
• Huge vocabulary
• Unseen words break it

The opposite idea is one token per character. The vocabulary is tiny, maybe a hundred symbols, and nothing is ever unknown. The cost is length. A sentence of forty words becomes about two hundred and fifty tokens, and the network must learn spelling before it can learn meaning. Long sequences are slow and expensive, so pure characters are rarely the best trade.

• Tiny vocabulary
• No unknown words
• Sequences get very long

Subword tokenization sits in the middle. Common words stay as one token, while rare words are cut into reusable pieces. Unbelievable might become un, believ, and able. The vocabulary stays manageable, the sequences stay short, and almost any new word can be assembled from known parts. This is what GPT-style models, BERT, and nearly every modern language model use.

• Frequent chunks stay whole
• Rare words split into pieces
• The modern default

Let us compare on one sentence of ten words. A word tokenizer gives ten tokens but needs a vocabulary of tens or hundreds of thousands. A character tokenizer gives around fifty tokens with a vocabulary of about a hundred. A subword tokenizer might give thirteen or fourteen tokens with a vocabulary around thirty to a hundred thousand. Balanced is the winning word here.

• Words: short, huge vocab
• Characters: long, tiny vocab
• Subwords: balanced

Byte-pair encoding learns its pieces from data with one simple rule. Start with every word split into characters. Count every adjacent pair of symbols across the corpus, weighted by word frequency. Merge the most frequent pair into a new symbol. Then repeat. Each merge adds exactly one entry to the vocabulary, so you stop when you reach the vocabulary size you want.

• Start from characters
• Find the most frequent adjacent pair
• Merge it, repeat

Our tiny corpus has four words. Low appears five times, lower two times, newest six times, and widest three times. We split each into characters and add an end of word marker, so low becomes l, o, w, end. The starting vocabulary has eleven symbols: l, o, w, e, r, n, s, t, i, d, and the end marker.

• low x5, lower x2
• newest x6, widest x3
• End marker: </w>

Now count pairs, weighted by frequency. The pair e, s appears in newest, six times, and in widest, three times, so nine. The pair s, t is also nine, and t followed by the end marker is nine as well. The pair l, o appears in low and lower, five plus two, so seven. The pair o, w is also seven. Nothing beats nine.

• e s: 6 + 3 = 9
• s t: 9, t end: 9
• l o: 7, o w: 7

Three pairs tie at nine, so we break the tie by taking the first one found: e and s. Merge one creates the symbol es. We recount. Now es followed by t is nine, so merge two creates est. Recount again: est followed by the end marker is nine, so merge three creates est end. Notice that the suffix est has become one reusable piece shared by newest and widest.

• Merge 1: e + s gives es (9)
• Merge 2: es + t gives est (9)
• Merge 3: est + end gives est</w> (9)

With the suffix handled, the best pair is now l, o with a count of seven, tied with o, w. We merge l and o into lo. Then lo followed by w is also seven, so merge five creates low. The word low is now a single token in both low and lower. The vocabulary has grown from eleven symbols to sixteen, five merges later.

• Merge 4: l + o gives lo (7)
• Merge 5: lo + w gives low (7)
• Now low is one token

Now we use the merges. Take the word lowest, which is not in our corpus. Split it into l, o, w, e, s, t, end. Apply merge one: e and s become es. Merge two gives est, merge three gives est end. Then merge four and five build low. The result is two tokens: low, and est end. A word we never saw is built from pieces we know. That is the whole trick.

• Apply merges in learned order
• lowest becomes low + est</w>
• Never seen, still handled

Three mistakes to avoid. First, merge order matters: at encoding time you must replay the merges in the same order they were learned. Second, count pairs weighted by word frequency, not by distinct words. Third, the tokenizer is trained once on a corpus and then frozen. If you change it after the model is trained, every id shifts meaning, and the model breaks.

• Order of merges matters
• Counts are weighted by frequency
• Tokenizer is trained once, then frozen

How many merges should you do? More merges give a bigger vocabulary and shorter sequences, but they make the embedding table larger and rare tokens get little training signal. Fewer merges give longer sequences. Real models commonly sit between about thirty thousand and a few hundred thousand tokens. It is a tuning choice, balancing sequence length against table size.

• Bigger: shorter sequences
• Bigger: larger embedding table
• Typical range: tens of thousands

Besides learned pieces, every vocabulary reserves a few special tokens. A padding token fills short sequences so a batch has equal length. Start and end tokens mark where text begins and stops. A separator splits two segments, and an unknown token stands for anything unrecognized. These get ordinary ids, but they carry jobs that the training process teaches the model to respect.

• Padding, start, end
• Separator and unknown
• Reserved ids with fixed jobs

What if the input contains a character the vocabulary never saw, like a rare emoji? A word-level system maps it to the unknown token, and the information is lost. Byte-level BPE avoids this by starting from the two hundred fifty six possible byte values instead of characters. Any text in any language is a sequence of bytes, so every input can be encoded, though rare symbols may take several tokens.

• Word models use an unknown token
• Byte-level BPE starts from 256 bytes
• Nothing is ever unknown

Two other names you will meet. WordPiece, used by BERT, also merges pieces step by step, but it picks the pair that most improves the likelihood of the training data, roughly the pair count divided by the counts of its two parts. It marks continuation pieces with a double hash. SentencePiece is a toolkit that treats text as a raw stream, with the space as an ordinary symbol, so it needs no language specific word splitting. It can train BPE or a unigram model.

• WordPiece: likelihood-style merge score
• Continuation pieces marked with ##
• SentencePiece: raw text, spaces as symbol

Now we have ids. Next we turn each id into a vector. The simplest vector is one-hot: as long as the vocabulary, with a one at the token's index and zeros elsewhere. With a toy vocabulary of five tokens, token number two is zero, zero, one, zero, zero. One-hot vectors are exact, but they are huge, and every pair of tokens is equally far apart. They carry no notion of similarity.

• Vector length = vocabulary size
• A single 1 at the token's index
• Everything else 0

Let E be a matrix with five rows, one per token, and three columns, the embedding size. Say row two is zero point nine, negative zero point one, zero point four. Multiply the one-hot vector zero, zero, one, zero, zero by E. Each output entry is a sum where only the row two term survives, so the result is exactly row two: zero point nine, negative zero point one, zero point four.

• E has one row per token
• one-hot times E picks a row
• Row 2 of E is the answer

Multiplying by a mostly zero vector wastes enormous effort, so in practice we skip the multiplication and index row two directly. The result is identical. Gradients are identical too: only the rows actually used in a batch receive updates, and every other row gets zero gradient. An embedding layer in any library is exactly this lookup.

• Matrix multiply wastes work
• Index directly instead
• Same result, same gradients

The embedding matrix is just a set of parameters, trained by backpropagation like any weights. Start with random numbers, and the loss gradually moves each row so that tokens used in similar ways end up with similar vectors. The size is vocabulary times dimension. With fifty thousand tokens and seven hundred sixty eight dimensions, that is thirty eight point four million numbers in this one table.

• Table size = vocab x dimension
• 50,000 x 768 = 38.4 million
• Trained by backpropagation

How should the rows get their meaning? A famous idea says a word is known by the company it keeps. Words that appear in similar contexts, like coffee and tea, should have similar vectors. Word2vec turned this into a training task. In the skip-gram version, you take a center word and train its vector to predict the words that appear around it.

• Words in similar contexts have similar meanings
• Predict neighbors to learn vectors
• Word2vec, skip-gram variant

Predicting over the entire vocabulary with softmax is expensive, so negative sampling simplifies it. For a real pair of center word c and context word o, we want the sigmoid of their dot product to be near one. For a few randomly drawn fake words k, we want the sigmoid of the negative dot product near one. The loss adds the negative log of each of those terms.

• Score = dot product of two vectors
• Push real pairs up with sigmoid
• Push random pairs down

Concrete numbers. Suppose the real pair has a dot product of two. The sigmoid of two is about zero point eight eight one, and negative log of that is zero point one two seven. Take one negative word whose dot product with the center is negative one. The sigmoid of one is zero point seven three one, and its negative log is zero point three one three. Total loss: zero point four four zero. Gradients then raise the first dot and lower the second.

• Real pair dot = 2: sigma = 0.881
• Negative dot = -1 gives 0.731
• Loss = 0.127 + 0.313 = 0.440

To compare embeddings we usually use cosine similarity: the dot product divided by the product of the two lengths. Geometrically it is the cosine of the angle between the arrows. A value near one means they point the same way, zero means unrelated, and negative one means opposite. Dividing by lengths matters, because frequent words can have long vectors that would otherwise dominate.

• cos = dot / (length x length)
• Measures angle, ignores length
• Range from -1 to 1

Take a equals two, one, zero and b equals four, two, one. The dot product is eight plus two plus zero, which is ten. The length of a is the square root of five, about two point two three six. The length of b is the square root of twenty one, about four point five eight three. Ten divided by about ten point two four seven gives zero point nine seven six. Now c equals zero, one, three: the dot with a is one, the length of c is about three point one six two, and the cosine is only zero point one four one.

• a = (2, 1, 0), b = (4, 2, 1)
• dot = 10, lengths 2.236 and 4.583
• cosine = 0.976

You may have heard that king minus man plus woman lands near queen. It sometimes does, and it shows that directions in the space can carry meaning. But there are caveats. The nearest result is often the input word itself, so demos usually exclude the inputs from the search. Results are far less tidy across many relations. And embeddings absorb the biases in their training text, so treat the analogy as a curiosity, not a guarantee.

• king - man + woman near queen
• Search usually excludes the inputs
• Embeddings also absorb data bias

The embedding lookup has a gap. The token dog gets the same vector whether it is the first word or the fifth. But dog bites man and man bites dog mean very different things. The model we build in the next lessons will process all tokens together, so unlike an RNN it has no built in sense of order. We must inject position into the vectors ourselves.

• Lookup ignores position
• Dog bites man is not man bites dog
• Position needs its own signal

The original transformer used fixed sine and cosine waves. For position pos and a dimension pair index i in a model of size d, the angle is pos divided by ten thousand to the power two i over d. Even dimensions use the sine of that angle, and odd dimensions use the cosine. Small i gives fast waves, large i gives very slow waves, so each pair oscillates at its own speed.

• Even dims: sin(pos / 10000^(2i/d))
• Odd dims: cos of the same angle
• Each pair has its own wavelength

Let d be four. Pair zero divides by one. Pair one divides by ten thousand to the power two over four, which is one hundred. At position zero we get sine zero, cosine zero, sine zero, cosine zero: zero, one, zero, one. At position one: sine of one is zero point eight four one, cosine of one is zero point five four zero, then sine of zero point zero one is about zero point zero one zero, cosine nearly one. At position two: zero point nine zero nine, negative zero point four one six, zero point zero two zero, and one.

• pos 0: 0, 1, 0, 1
• pos 1: 0.841, 0.540, 0.010, 1.000
• pos 2: 0.909, -0.416, 0.020, 1.000

Why this design? Fast pairs distinguish neighbors, slow pairs distinguish far apart positions, much like the digits of an odometer turning at different rates, so each position gets a unique fingerprint. Also, moving from position pos to pos plus k rotates each sine cosine pair by a fixed angle, so relative offsets look the same everywhere. We simply add this vector to the token embedding.

• Each position gets a unique pattern
• Shifting by k is a rotation
• Like digits of a counter at different speeds

Two alternatives, at a high level. Learned positional embeddings are another lookup table with one trainable row per position, simple but limited to a maximum length. Rotary embeddings rotate pairs of dimensions in the query and key vectors by an angle proportional to position, so the dot product between two tokens depends on how far apart they are. Rotary is widely used in recent models. We will see where it plugs in when we meet attention.

• Learned: a second lookup table
• Rotary: rotate vectors by position
• Relative offset appears in dot products

Now the code. From transformers import AutoTokenizer, then call from pretrained with a model name like gpt two. Calling the tokenizer on a sentence returns input ids. Convert ids to tokens to see the pieces, and decode to get the text back. Try a rare word like tokenization or a made up word and you will see it split into subwords. Check len of the tokenizer for the vocabulary size, and special tokens for the reserved ones.

• AutoTokenizer.from_pretrained
• encode gives ids, convert gives pieces
• decode reverses it

To embed, create torch nn Embedding with vocabulary size and dimension, then pass a tensor of ids. The output has one extra axis of size dimension. Common pitfalls: always use the tokenizer that matches the model, because ids are tied to its table. Watch padding and maximum length, since longer inputs get truncated. And remember that token counts, not word counts, determine cost and context limits.

• torch.nn.Embedding(vocab, dim)
• Use the model's own tokenizer
• Mind padding and max length

Let us recap. Text is split into subword tokens by an algorithm like byte-pair encoding, which repeatedly merges the most frequent pair, as we did with low, newest, and widest. Ids are addresses into an embedding matrix, which is one-hot times a matrix. Cosine similarity compares vectors, and sinusoidal or rotary methods add position. Next time, in lesson 55, we derive attention with numbers.

• Text to tokens to ids to vectors
• BPE merges frequent pairs
• Position added to embeddings