Sentence Embeddings and Contrastive Training: How Meaning Becomes Math

Search for, how do I reset my password, and a keyword engine hunts for those exact words. But the help article is titled, I forgot my login. Zero shared words, identical meaning. Today we fix that by teaching a model to turn text into points in space, where similar meanings land close together.

• Keyword search misses paraphrases
• Embeddings map meaning to points

This is lesson eleven. Last time we made prompts measurable with structured output and systematic evaluation. Now we begin the retrieval half of the course. By the end you will understand how sentence embeddings are built, derive the contrastive loss that trains them, compute it by hand, and implement it in PyTorch.

• Pooling and contrastive loss
• Hand-computed example
• PyTorch implementation

Recall from the ML course's embeddings lesson that every token gets a vector, a learned row in a lookup table. The word bank gets one starting vector. But a search system does not compare tokens, it compares whole questions, paragraphs and documents. We need one single vector for an entire piece of text.

• Tokens each get a vector
• Search needs one vector per text

The geometric intuition is simple. Imagine a map where every text is a pin. Texts about cooking cluster in one region, texts about tax law in another. A question and its answer should sit very near each other, while unrelated texts sit far apart. Search then becomes finding the nearest pins to your query pin.

• Similar texts sit near each other
• Search becomes nearest-neighbour lookup

Why not just average raw word vectors? That works a little, but it ignores context. The dog bit the man and the man bit the dog contain the same words, and the average is identical. A transformer encoder fixes this, because attention lets each token vector absorb its context before we combine them.

• Averaging raw words loses order
• Encoders add context via attention

An encoder like BERT outputs one contextual vector per token. If your text has twelve tokens and the hidden size is seven hundred sixty eight, you get a twelve by seven hundred sixty eight grid. Pooling is the step that collapses that grid into a single vector of length seven hundred sixty eight.

• Encoder gives one vector per token
• Pooling yields one vector per text

Mean pooling is the most common choice. Add up the token vectors and divide by the number of real tokens. Take a toy example with three tokens whose first coordinates are two, four and six. The mean is twelve over three, which is four. Padding tokens must be excluded using the attention mask, or short texts get dragged toward zero.

• Average token vectors
• Mask out padding tokens

The alternative is CLS pooling. BERT-style models prepend a special classification token, and during training that token learns to summarize the whole input. You simply take its output vector and ignore the rest. Which one is right depends on how the model was trained, so always follow the model card, never guess.

• Special token summarizes input
• Follow the model card

But here is the catch. A raw pretrained encoder, even with perfect pooling, gives mediocre sentence vectors. It was trained to fill in masked words, not to place similar sentences near each other. Distances between its vectors are not tuned for meaning. We need a training objective that shapes the geometry directly.

• Masked LM does not target similarity
• Geometry must be trained

Contrastive learning gives that objective. The idea is a party game. Show the model a positive pair, two texts that should match, like a question and its answer. Pull them together. Show it negatives, texts that do not match, and push them apart. Repeat that millions of times and the space organizes itself around meaning.

• Positives get pulled close
• Negatives get pushed away

Where do the negatives come from? The trick is in-batch negatives. Take a batch of three pairs. For the first question, its own answer is the positive, and the answers belonging to the other two questions act as free negatives. No extra labeling needed, and a bigger batch means more negatives for the same cost.

• Other pairs' documents are negatives
• Bigger batch, more negatives

Now the math. Let q be a query embedding, d plus its true positive document, and d one through d B all documents in the batch, including d plus. Let s of q and d be a similarity score, usually cosine similarity, a number between minus one and one. Tau, the temperature, is a small positive constant that we will explain shortly.

• q: query, d plus: true document
• s: similarity, tau: temperature

The loss is minus log of a fraction. The numerator is e to the power of s of q and d plus, divided by tau. The denominator is the sum over every document j in the batch of e to the power of s of q and d j, divided by tau. Written fully, L equals minus log of exp of s over tau for the positive, over the sum of exp of s over tau across all j.

• Minus log of a fraction
• Numerator: positive, denominator: all

Here is the key insight. That fraction is a softmax. It is the probability the model assigns to picking the correct document out of the batch. So InfoNCE is just cross-entropy, which you know from the ML course, applied to a classification problem where the classes are the documents and the correct class is the matching one.

• Fraction equals softmax probability
• Classes are the batch documents

Look at how the loss behaves. If the positive gets probability close to one, minus log is near zero, so the loss is tiny. If the positive gets probability one tenth, the loss is about two point three. Minimizing the loss raises the positive's score and lowers every negative's score at the same time.

• p near 1 gives loss near 0
• p of 0.1 gives loss 2.3

Let us compute it. Three pairs in a batch, so the similarity matrix is three by three. Rows are queries, columns are documents. Row one is zero point eight, zero point three, zero point one. Row two is zero point two, zero point seven, zero point four. Row three is zero, zero point three, zero point six. The diagonal holds the true pairs.

• Rows: queries, columns: documents
• Diagonal: true pairs

Pick temperature tau equals zero point five. Dividing by tau just doubles every score. Row one becomes one point six, zero point six, zero point two. Exponentiate: about four point nine five, one point eight two, and one point two two. They sum to about eight. The positive probability is four point nine five over eight, roughly zero point six two.

• Divide by tau: 1.6, 0.6, 0.2
• Softmax gives p of 0.62

The loss for row one is minus the natural log of zero point six two, which is about zero point four eight. Row two scores become zero point four, one point four, zero point eight. Exponentials are one point four nine, four point zero six, two point two three, summing to seven point seven seven. Probability zero point five two, loss about zero point six five.

• Row one loss: about 0.48
• Row two loss: about 0.65

Row three scores become zero, zero point six, one point two. Exponentials are one, one point eight two, three point three two, summing to six point one four. The positive probability is about zero point five four, so the loss is about zero point six two. Average the three losses: zero point four eight, plus zero point six five, plus zero point six two, divided by three, is about zero point five eight.

• Row three loss: about 0.62
• Batch loss: about 0.58

Now watch what temperature does. Lower tau to zero point one and the scores are multiplied by ten. Row one becomes eight, three, one. Now the positive takes over ninety nine percent of the probability, and the loss collapses to about zero point zero zero eight. Small tau sharpens the softmax and makes the loss focus on the hardest mistakes.

• Small tau sharpens the softmax
• Large tau flattens it

Contrastive training lives or dies on its pairs. Common sources include search logs, where a query is paired with the result users clicked. Natural language inference data, where entailment sentences form positives. And titles paired with abstracts or body text, which exist at web scale. Question and answer forums give another huge pool of pairs.

• Search logs, NLI, titles and abstracts
• Q and A pairs at web scale

Random in-batch negatives are easy: a cooking question versus a tax answer. After a while the model separates those trivially and learns little. Hard negatives are documents that look relevant but are wrong, like a password-reset article for a different product. They are often mined with a keyword search or an earlier model, and they teach fine distinctions.

• Easy negatives teach little
• Hard negatives sharpen boundaries

Hard negatives carry a danger: false negatives. If your mining picks a document that is actually relevant but unlabeled, you are training the model to push away a correct answer. Teams usually filter mined candidates, for example skipping those ranked too high, and they keep a mix of easy and hard so the signal stays clean.

• Unlabeled positives hurt training
• Filter and mix negatives

Batch size matters too. With in-batch negatives, a batch of thirty two gives each query thirty one negatives, while a batch of several thousand gives far more and a much harder classification problem. That is why large embedding runs favor big batches, often using tricks to fit them across many accelerators in memory.

• Negatives per query: batch minus one
• Bigger batches cost more memory

Queries and documents are not the same kind of text. A query is short and vague, a document is long and detailed. Many modern models are asymmetric and expect prefixes, such as query colon before a question and passage colon before a document. The prefix tells the encoder which role the text plays, and skipping it quietly hurts retrieval quality.

• Short query, long document
• Use the prefixes the model expects

Another idea is Matryoshka embeddings, named after the nesting dolls. The model is trained so the first sixty four numbers already form a decent embedding, the first two hundred fifty six a better one, and the full vector the best. You can truncate the vector to save storage and speed, trading a little accuracy for a lot of size.

• Prefixes of the vector still work
• Truncate to trade accuracy for size

Know the limits too. Multilingual models map many languages into a shared space, so a French question can find an English answer, but quality is usually uneven across languages. And domain shift is real: a model trained on web text may blur the difference between terms that are distinct in medicine, law or your own product codes. Always test on your own data.

• Quality varies by language
• Test on your own domain data

Time for code. With the sentence-transformers library, you load a model by name, call encode on a list of strings, and get back a matrix with one row per text. Then you compare vectors with cosine similarity. Pick any embedding model your provider or the open source hub offers, and read its card for prefixes and max length.

• encode returns one row per text
• Check prefixes and max length

If you prefer plain Hugging Face, load the tokenizer and model, run the batch, then do mean pooling yourself. Multiply the hidden states by the attention mask, sum along the token axis, and divide by the mask sum. Then normalize to unit length. Those few lines are exactly what the high level library does for you.

• Mask, sum, divide by count
• Normalize to unit length

Now the loss. Given query embeddings and document embeddings, both normalized, compute the similarity matrix with a matrix multiply, divide by tau, and call cross-entropy with labels zero, one, two, up to the batch size minus one. The correct document for query i sits in column i. That is the whole InfoNCE loss in four lines.

• Matrix multiply gives all scores
• Labels are 0 through B minus 1

Many implementations also add the symmetric loss: compute cross-entropy over rows, then over columns, and average. That trains documents to find their queries as well. Also remember that gradients flow into both sides, so the encoder gets updated from every pair in the batch, positives and negatives alike.

• Rows: query to document
• Columns: document to query

Mistake one: mixing embedding models. Each model has its own private coordinate system, so vectors from two models are not comparable, even when both have the same dimension. If you index documents with one model, you must embed queries with exactly that model and version. Upgrading the model means re-embedding everything.

• Each model has its own space
• Same model for index and query

Mistake two: ignoring max token length. Many encoders accept only a limited number of tokens, such as five hundred twelve, and silently truncate the rest. A long document whose key facts come late gets embedded from only its opening. Check the limit, count tokens, and split long text into chunks, which is exactly where our RAG lessons are heading.

• Encoders truncate silently
• Count tokens and chunk text

A few more traps. Forgetting the padding mask in mean pooling. Using a temperature copied from another setup without tuning it. Skipping prefixes. And treating a similarity score as a probability: a score of zero point eight means different things in different models. Compare scores within one model, and evaluate with real queries.

• Mask padding, tune tau
• Scores are model-specific

Let us recap. Pooling collapses token vectors into one text vector. Contrastive training pulls matching pairs together and pushes others apart. InfoNCE is cross-entropy over in-batch negatives, with temperature controlling sharpness. Good data and hard negatives drive quality. Prefixes, Matryoshka truncation and model consistency matter in practice.

• Pooling, contrastive loss, InfoNCE
• Data, prefixes, model consistency

We keep saying nearest, but nearest by what measure? Next lesson compares cosine similarity and dot product, shows when they agree and when they do not, and builds exact nearest neighbour search.

• Cosine versus dot product
• Exact nearest neighbour search