Product Quantization: Shrink a 3072-Byte Vector to 96 Bytes

One embedding takes 3072 bytes. Product quantization stores it in just 96.

• 32x smaller

Take a vector with 768 numbers. Each is a 32-bit float, four bytes. Seven hundred sixty-eight times four is 3072 bytes.

• 768 dimensions
• 4 bytes each
• 3072 bytes total

Now chop it into 96 chunks of 8 numbers each.

• 96 chunks
• 8 numbers each

For each chunk position, k-means learns a codebook of 256 centroids.

• k-means per chunk position
• 256 centroids each

Replace each chunk with the index of its nearest centroid. Two hundred fifty-six options fit in one byte.

• Nearest centroid wins
• Index fits in 1 byte

So the vector becomes 96 bytes. That is 32 times smaller. A billion vectors drop from about three terabytes to ninety-six gigabytes.

• 96 bytes per vector
• 3 TB down to 96 GB

The catch: it is lossy. Distances become approximate, so recall drops. Reranking top candidates with full vectors wins much of it back.

• Lossy compression
• Rerank to recover recall

Next lesson: ANN indexes, where this trick meets IVF and HNSW. Follow so you don't miss it.

• IVF
• HNSW
• Follow for more