Cosine vs Dot Product: How Exact Vector Search Really Works

Imagine your embedding model is excellent, your documents are clean, and yet your search keeps returning odd results. Very often the culprit is not the model at all. It is a quiet mismatch between how the vectors were trained to be compared and how your index actually compares them.

• Right model, wrong results
• The metric is the culprit

In the last episode we saw how contrastive training turns sentences into vectors, pulling similar meanings together. Today we answer the next question: given those vectors, how do we measure closeness, and how do we find the nearest ones exactly, without any shortcuts?

• Last time: contrastive training
• Today: comparing vectors, exactly

We will define dot product, cosine similarity and Euclidean distance, then derive how they connect. We will rank three vectors where the measures disagree, price out brute-force search for ten million vectors, and meet the curse of dimensionality. Finally we set up recall at k, the yardstick for everything that follows.

• Three measures and one identity
• Brute-force search and its cost
• Recall as the yardstick

The dot product of two vectors multiplies matching coordinates and adds them up. For a equals one, two and b equals three, four, that is one times three plus two times four, which is eleven. Geometrically it equals the length of a, times the length of b, times the cosine of the angle between them.

• a · b = sum of a_i b_i
• Also |a||b| cos theta

Here is the intuition. Drop a shadow of b onto the direction of a, then multiply that shadow's length by the length of a. A long vector pointing roughly the same way gets a big score. So the dot product rewards both alignment and sheer length, and that second part will matter a lot.

• Project one vector onto the other
• Length and direction both matter

Cosine similarity divides the dot product by both lengths. That strips out magnitude and leaves only the angle. The result lives between minus one and one: one means the same direction, zero means perpendicular, and minus one means opposite. Doubling a vector changes its dot product, but its cosine stays exactly the same.

• cos = a · b / (|a||b|)
• Range minus one to one
• Ignores length

Euclidean distance is the straight-line gap between the tips of two vectors. Subtract coordinate by coordinate, square, add, and take the square root. Unlike the other two, smaller means more similar. It is sensitive to both direction and length, so it behaves differently again.

• ||a − b|| = sqrt(sum (a_i − b_i)^2)
• Straight-line distance

A practical detail trips people up. Dot product and cosine are similarities, where higher is better. Euclidean distance is a distance, where lower is better. Libraries hide this by negating scores or sorting in different directions, so always check which way your index ranks before trusting a result list.

• Higher is better: dot, cosine
• Lower is better: Euclidean
• Flip signs carefully

Normalizing a vector means dividing it by its own length, so the result has length exactly one. Take three, four. Its length is five, so the unit version is zero point six, zero point eight. All normalized vectors lie on the surface of a sphere, and only their direction remains.

• Normalize: divide by length
• Every vector has length one

Now the key derivation. Expand the squared distance: the length of a minus b squared equals length of a squared, minus two times a dot b, plus length of b squared. If both vectors have length one, the first and last terms are each one. So squared distance equals two minus two times the cosine.

• ||a−b||² = ||a||² − 2a·b + ||b||²
• Unit vectors: 1 − 2cos + 1

Read that result carefully. As cosine goes up, squared distance goes down, in a perfectly steady relationship. So for unit vectors, sorting by highest cosine, highest dot product, or smallest Euclidean distance produces exactly the same order. Three metrics collapse into one, which is why normalization is such a powerful simplification.

• Distance shrinks as cosine grows
• Same ranking for all three
• Monotonic relationship

Let us sanity check with numbers. If two unit vectors have cosine zero point nine five, squared distance is two minus one point nine, which is zero point one. Perpendicular unit vectors have cosine zero, so squared distance is two. And opposite vectors give four, the maximum possible on the sphere.

• cos 0.95 gives d² = 0.10
• cos 0 gives d² = 2

Time for a worked example in two dimensions. Our query is q equals two, one. Document A is four, two, which points exactly the same way but is longer. Document B is ten, ten, a very long vector pointing somewhat off. Document C is zero, three, a short vector pointing well away.

• Query q = (2, 1)
• A = (4, 2), B = (10, 10), C = (0, 3)

Dot products first. Q dot A is two times four plus one times two, which is ten. Q dot B is twenty plus ten, thirty. Q dot C is zero plus three, three. Ranked by dot product, B wins by a mile, then A, then C. The long vector B is simply louder.

• q·A = 10
• q·B = 30
• q·C = 3

Now cosine. The length of q is root five, about two point two three six. A has length four point four seven two, so its cosine is ten divided by ten, exactly one. B has length fourteen point one four two, giving thirty over thirty one point six two, or zero point nine four nine. C gives three over six point seven, about zero point four four seven.

• |q| = 2.236
• cos A = 1.000
• cos B = 0.949
• cos C = 0.447

Here is the punchline. Dot product says B, A, C. Cosine says A, B, C. Euclidean distance says A, then C, then B, because B sits far away, about twelve units from q, while C is only two point eight away. Three metrics, three different orderings, from the very same vectors.

• Dot: B, A, C
• Cosine: A, B, C
• Euclidean: A, C, B

Now normalize every vector first. A and q point the same way, so their squared distance is zero. B has cosine zero point nine four nine, so squared distance is about zero point one. C has cosine zero point four four seven, so squared distance is about one point one. Every metric now agrees: A, then B, then C.

• Unit versions of A, B, C
• All metrics agree: A, B, C

So which should you use? If vector length is mostly an accident, such as an artifact of text length or token frequency, cosine is the safer choice because it looks only at direction. Most popular sentence embedding models are intended for exactly this case, and many are already trained to output unit-length vectors.

• Text length can inflate norms
• Cosine focuses on meaning

But sometimes length carries information. In some retrieval and recommendation models, a larger norm can reflect popularity or confidence, and the model is trained so that the dot product is the score. If you normalize those vectors, you throw away something the model deliberately learned to use.

• Norm can encode popularity or confidence
• Recommenders often use dot product

The rule of thumb is simple: search with the same similarity the model was trained with. Many contrastive recipes compute cosine similarity between pairs, scale it by a temperature, and feed it to a softmax cross-entropy over the batch, as in the last episode. Such a model is optimized for angles, so cosine or normalized dot product fits it. Always check the model card.

• Train with cosine, search with cosine
• Temperature scales the logits
• Check the model card

Here is a quick decision table. If the vectors are already unit length, all three metrics give the same ranking, so pick the fastest, usually inner product. If the vectors are not normalized and the model wants direction, normalize and use inner product. If the model was trained with raw dot product, keep it raw.

• Unit vectors: all equivalent
• Unnormalized, direction only: cosine
• Trained for dot: dot product

Exact k nearest neighbour search is conceptually simple. Compute the score between the query and every stored vector, then keep the top k. With N vectors of dimension d, that is N times d multiplications per query. No index, no cleverness, and the answer is guaranteed to be the true nearest neighbours.

• Score every vector
• Keep the best k
• Cost: O(N · d)

The trick is to do all of this as one matrix multiplication. Stack the documents into a matrix D with N rows, and your queries into a matrix Q with B rows. Then Q times D transposed gives a B by N score matrix in one call. Optimized linear algebra libraries, and GPUs especially, are extremely fast at exactly this.

• Scores = Q × Dᵀ
• Batch queries together
• GPUs and BLAS love this

Once you have N scores, do not sort them all. A full sort costs N log N, but you only need the best k. A partial selection, such as numpy's argpartition or torch's topk, finds the top k in roughly linear time. Then sort just those k results for display.

• Full sort: O(N log N)
• Partial selection: O(N)
• argpartition, torch.topk

In code this is just a few lines. Normalize documents once at index time, normalize the query, compute scores with a matrix product, and call topk. In NumPy that is the at operator followed by argpartition. In PyTorch it is the same product followed by torch dot topk, on CPU or GPU, in float32 or half precision.

• Normalize both sides
• scores = q @ D.T
• topk on scores

Now the memory math. Ten million vectors, 768 dimensions each, four bytes per float32 number. That is ten million times 768, which is seven point six eight billion numbers, times four bytes, giving thirty point seven two billion bytes, about thirty gigabytes. That is before any metadata or the text itself.

• 10,000,000 × 768 × 4 bytes
• = 30.72 GB

What about compute? Each query needs about two times N times d operations, counting a multiply and an add. That is two times ten million times 768, roughly fifteen billion operations. Modern hardware can do that, but every query also has to stream the whole thirty gigabyte matrix through memory, and memory bandwidth, not arithmetic, usually becomes the bottleneck.

• 2 · N · d ≈ 15.4 GFLOPs
• Bandwidth-bound in practice

There are levers. Storing vectors in half precision cuts memory to fifteen point three six gigabytes. Eight-bit integers cut it to seven point six eight, at some cost in accuracy. And batching queries means one pass over the data serves many queries at once, which raises throughput even though each pass still touches everything.

• float16: 15.36 GB
• int8: 7.68 GB
• Batching amortizes reads

High dimensions behave strangely. Take two random directions in 768 dimensions. Their cosine is almost always close to zero, with a typical spread of about one over root d, roughly zero point zero three six. Distances between points bunch together, so the nearest neighbour is often only slightly nearer than a typical point.

• Random vectors look alike in high d
• Cosine spread about 1/√d
• d = 768: about 0.036

This is why classic spatial trees, which work well in two or three dimensions, lose their advantage at hundreds of dimensions. They end up visiting most of the data anyway. Meanwhile brute force cost grows linearly with N. At millions of vectors and strict latency targets, exact search stops being practical, and we start accepting approximate answers.

• Classic tree indexes degrade in high d
• Cost grows linearly with N
• Trade a little accuracy for speed

If we accept approximate answers, we need a way to measure the damage. Recall at k does that. Run exact search to get the true top k, run the approximate method, and count how many true neighbours it found, divided by k. If it returns nine of the true ten, recall at ten is zero point nine.

• Exact top-k is ground truth
• recall@k = overlap / k

Brute force never goes away. It is the ground truth for tuning approximate indexes. And for corpora up to perhaps hundreds of thousands of vectors, it is often fast enough, simpler, and perfectly accurate. Do not reach for a complicated index until measurement shows you need one.

• Ground truth for tuning
• Best choice for small corpora
• Exact baseline for tests

Let us benchmark it yourself. Generate random float32 vectors with a fixed seed, normalize them, and for several values of N, time a batch of queries through the matrix product and top-k. Warm up first, run several repeats, and on a GPU synchronize before stopping the clock. I will not quote numbers, because they depend entirely on your hardware.

• Random float32 vectors
• Vary N, fixed d = 768
• Time matmul plus top-k

When you plot the results, expect time to grow roughly in proportion to N. If it jumps sharply, you may have run out of cache or RAM, or started swapping. Compare against your hardware's memory bandwidth to see how close you are to the limit. This is the baseline every approximate index in the next episodes must beat.

• Time should grow roughly linearly in N
• Check memory vs compute limits

Now the common mistakes. First, forgetting to normalize. If you use a raw inner product index on unnormalized vectors, long vectors dominate, exactly like document B in our example. The fix is to normalize both documents and queries consistently, or to choose an index metric that handles it.

• Raw dot product is not cosine
• Long vectors dominate results
• Normalize at index and query time

Second, mixing metrics between model and index. Many libraries default to Euclidean distance, while your model may expect cosine. On unit vectors that is harmless, thanks to our identity, but on unnormalized vectors it silently changes the ranking. Also be careful comparing scores across metrics, since their ranges and directions differ.

• Index metric must match the model
• Inner product vs L2 defaults
• Document the choice

Let us recap. Dot product mixes angle and length, cosine keeps only angle, and Euclidean distance measures the gap. For unit vectors, squared distance equals two minus two cosine, so all three agree. Match the metric to training. Brute force costs N times d per query and thirty point seven two gigabytes for ten million vectors. Recall at k will judge every shortcut.

• Unit vectors: ||a−b||² = 2 − 2cos
• Match metric to training
• Brute force: O(N·d), 30.72 GB at 10M×768
• Recall@k measures approximation

In the next episode we escape the linear scan with approximate nearest neighbour indexes: HNSW graphs, IVF clustering, and product quantization.

• HNSW graphs
• IVF clustering
• Product quantization