Vector Databases in Practice: Filtering, Sizing, and Real Code

You built an index that finds the nearest vectors among millions in a few milliseconds. Great. Now your server restarts, a document changes, and a customer asks you to delete all of their data. The index knows nothing about any of it. Today we turn a clever index into a real system.

• Fast search is only the start
• Restarts, edits, deletions
• Today: index to real system

Last time, in the ANN lesson, we studied HNSW, IVF and product quantization, the algorithms that make nearest neighbour search fast. This is lesson fourteen, and it closes the embeddings section. We will cover what a vector database adds, the landscape, filtering, multi-tenancy, sizing with real numbers, runnable Python, and the mistakes that bite in production.

• Recall: HNSW, IVF, PQ
• Filtering, tenancy, sizing
• Runnable Python

An index answers exactly one question: given a query vector, which stored vectors are nearest? That is all. A FAISS index in memory is a data structure, not a service. If the process dies, it vanishes unless you serialized it yourself, and nothing tells you which vector belongs to which document.

• Query vector in
• Nearest vectors out
• Nothing more

A vector database wraps the index with what we expect from any database. Persistence, so data survives restarts. CRUD, meaning create, read, update and delete by identifier. Metadata stored beside each vector, and filtering on it. Then, at larger scale, replication for availability and sharding to spread data across machines.

• Persistence
• Create, read, update, delete
• Metadata and filtering
• Replication and sharding

Think of the index as the engine and the database as the whole car. The engine is the hard part, but you cannot commute with just an engine. You also need the chassis, which is storage, the steering wheel, which is the query API, and the seatbelts, which are backups and access control.

• Index is the engine
• Database is the car
• Storage, API, backups, access

Each record typically has three parts: an identifier that you control, the vector itself, and a payload of metadata, often including the original text or a pointer to it. A query returns ids, distances, and optionally payloads. Note that most databases store the vectors you hand them. They do not produce embeddings unless you configure a model.

• Id you control
• Vector
• Metadata payload

The landscape falls into a few families, and I will not rank them, because the right choice depends on your workload. First, dedicated vector databases, built from the ground up around vector search. They usually offer sharding, replication and rich filtering, and they are designed to scale as their own service.

• Built around vector search
• Sharding, replication
• Rich filtering

Second, extensions to databases you already run. The best known is pgvector for Postgres, which adds a vector column type and vector index types. The big advantage is that vectors live next to your relational data, so one query and one transaction can cover both the business data and the similarity search.

• pgvector for Postgres
• Vectors beside relational data
• One transaction, one join

Third, search engines that added vector support. If you already run a full text engine, you can often store vectors in the same documents and combine keyword scoring with vector similarity. That is a preview of hybrid search, which we will build properly later in this course.

• Full-text engines added vectors
• Keyword plus vector scoring
• Preview of hybrid search

Fourth, embedded options such as Chroma and LanceDB. These run inside your Python process, or beside it, on local files, with no server to operate. They are ideal for prototypes, notebooks, and small to medium applications. FAISS is closer to a pure index library, so you build persistence and metadata handling yourself.

• Chroma, LanceDB
• FAISS is a library
• No server needed

How do you choose? Ask five questions. How many vectors do you have, now and in a year? How fast do they change? How heavy are your filters? Do you need transactions with other data? And what can your team realistically operate? One million vectors next to an existing Postgres is a very different answer than a billion vectors across regions.

• How many vectors?
• How fast do they change?
• How heavy are filters?
• Need joins or transactions?
• What can your team run?

Start with the schema. The embedding dimension is fixed by your model, say seven hundred sixty eight. Pick the distance metric, cosine, dot product or Euclidean, to match how the model was trained, as in the previous lessons. Then decide the metadata fields, such as source, language, creation date, and tenant.

• Fixed dimension, e.g. 768
• Metric matches the model
• Typed metadata fields

Keep metadata purposeful. Fields you filter on should be typed and indexed. Very large documents often live elsewhere, with just an identifier stored in the record. And store the embedding model name and version on every record. That single field will save you later, as we will see.

• Index fields you filter on
• Keep big blobs elsewhere
• Store model name and version

Semantic search alone is rarely enough. Users want documents from last year, in French, from one department. That is a filtered query: nearest neighbours among only those vectors whose metadata matches a condition. How the engine combines the two steps matters enormously for results.

• Nearest among matching vectors
• Users expect it
• How you combine matters

Post-filtering searches first, then discards non-matching results. Worked example: one million vectors, and your filter matches one percent, which is ten thousand vectors. Ask for the top ten, then filter. On average only one tenth of one result survives. Very often you get zero results, even though ten thousand valid ones exist.

• 1,000,000 vectors
• Filter matches 1% = 10,000
• Top 10, then filter: ~0.1 survive

The fix people reach for is over-fetching: request more candidates, say a thousand, then filter down. At one percent selectivity, a thousand candidates yields about ten matches on average. But it costs latency, and if the filter is rarer, you must fetch even more. It becomes guesswork instead of engineering.

• Fetch 1,000 candidates
• 1% pass: about 10
• Costs latency, still guesswork

Pre-filtering applies the filter first, then searches only the allowed vectors. If the allowed set is small, the engine can simply brute force it exactly, which is fast and perfectly accurate. For one percent of a million, that is ten thousand distance computations, which is trivial on modern hardware.

• Filter first, then search
• Small set: exact brute force
• 10,000 comparisons: trivial

But with a graph index like HNSW, naive pre-filtering has a pitfall. If you remove the nodes that do not match, the remaining graph can become disconnected, and the search gets stuck in one region and returns poor results. Good engines check the filter during graph traversal, or switch strategies depending on how selective the filter is.

• Removing nodes cuts edges
• Search gets stuck
• Good engines filter during traversal

My practical advice: test filtered recall yourself. Compare the approximate filtered results to brute force exact results on a sample of queries, using realistic filters. Do not trust unfiltered benchmarks. And index the metadata fields you filter on, otherwise the filter itself becomes the slow part.

• Compare against exact search
• Use realistic filters
• Index filtered fields

Now multi-tenancy: many customers in one system, and one customer must never see another's data. You have three main options. A separate collection per tenant, a namespace or partition per tenant, or one shared collection with a tenant identifier field applied as a filter.

• Collection per tenant
• Namespace or partition
• Shared, tenant id filter

The trade-offs are real. Separate collections give strong isolation and easy tenant deletion, but thousands of small indexes cost overhead. A shared collection is efficient, but one forgotten filter leaks data. Namespaces sit in between. Whatever you pick, enforce the tenant filter on the server side, and never trust client code to remember it.

• Separate: strong isolation, overhead
• Shared: efficient, leak risk
• Enforce filter server-side

Now writes. Upsert means insert if the id is new, replace if it already exists. Use stable ids, such as a hash of the document source and the chunk position, so re-ingesting a document overwrites instead of duplicating. Random ids on every ingestion run are how you end up with five copies of every paragraph.

• Insert new, replace existing
• Id = hash(source, chunk position)
• Re-ingest without duplicates

Deletes are subtler. In many indexes, particularly graphs, a delete just marks a vector as removed, a tombstone, and space is reclaimed later by compaction. So after heavy deletes, memory may not drop immediately. Also, when a document changes, delete all of its old chunks by a source filter, because the new version may have fewer chunks than the old one.

• Tombstones, then compaction
• Memory may not drop at once
• Delete old chunks by source id

Embedding models change. A vector from model A and a vector from model B live in different spaces, even when the dimensions are identical. Comparing them gives meaningless scores. So you cannot mix them in one index, and changing models means re-embedding everything.

• Different model, different space
• Same dimension, meaningless scores
• Re-embed everything

A safe pattern: build a new collection named with the new model version, backfill it, run your evaluation queries against both, then switch traffic by flipping an alias, and keep the old collection for rollback. Query embeddings must come from the same model version as the stored ones, which is exactly why we stored the version on every record.

• New collection per model version
• Backfill, then evaluate both
• Flip an alias, keep rollback

Now sizing, with numbers. Suppose ten million chunks, each embedded as seven hundred sixty eight dimensions in thirty two bit floats. Each float is four bytes, so one vector is three thousand seventy two bytes. Multiply by ten million and you get about thirty point seven gigabytes of raw vectors.

• 10,000,000 vectors
• 768 dims x 4 bytes = 3,072 B
• Total: about 30.7 GB

Add the index. An HNSW graph with sixteen links per node adds roughly one hundred to two hundred bytes per vector, so about one to two gigabytes here. Add metadata at around five hundred bytes per record, which is five gigabytes. The total is roughly thirty seven to forty gigabytes. These are estimates, because real overhead varies by engine.

• HNSW links: ~1.3 to 2 GB
• Metadata ~500 B each: 5 GB
• Total about 37 to 40 GB

Replication multiplies it. A primary plus two replicas is three copies, so well over a hundred gigabytes if everything sits in RAM. That is costly. The levers: store vectors as eight bit integers, a quarter of the size, about seven point seven gigabytes. Or use product quantization at ninety six bytes per vector, under one gigabyte, trading some recall. Or keep full vectors on disk, compressed ones in memory, and rerank.

• Three copies: 100+ GB
• int8: quarter size, 7.7 GB
• PQ at 96 B: under 1 GB

Dimension is a direct lever too. Halving the dimension halves the memory. Some embedding models support truncating vectors to fewer dimensions with a small quality loss, but only if they were trained for it. Check your model's documentation, and measure on your own queries before committing.

• Half the dimensions, half the memory
• Only if the model supports truncation
• Measure on your own queries

Let's build it, using Chroma as a local embedded store, though the same steps apply anywhere. Install the library, create a persistent client pointing at a folder, and get or create a collection. We configure cosine distance for the collection, and we use a sentence embedding model to produce the vectors.

• Persistent client, local folder
• Get or create collection
• Cosine distance

Add the documents. We call upsert with lists of ids, texts, embeddings and metadata. For example, three support articles with ids doc one to doc three, each with metadata such as language and year, plus an embedding model field holding the model name and version. Because the client is persistent, a restart keeps everything.

• ids, documents, embeddings
• metadata: language, year, model
• Survives restarts

Now query. Embed the question with the same model, then call query with the query embedding, three results, and a where clause such as language equals English and year at least twenty twenty four. Back come ids, distances and metadata. Note that a lower distance is closer, and cosine distance is one minus cosine similarity.

• Embed the question, same model
• where: language and year
• Lower distance is closer

To update, call upsert with the same id and the new text, new embedding and new metadata. To delete, call delete with ids, or with a where filter, such as every record from a retired source. Then call count to confirm. That is the full lifecycle: create, add, filtered query, update, delete.

• Update: upsert same id
• Delete by ids or by filter
• Count to confirm

Real uses. Semantic search: users find the help article by meaning rather than exact words. Deduplication: embed items, find pairs whose cosine similarity exceeds a threshold such as point nine five, and tune that threshold on labelled examples. Recommendations: use an item a user liked as the query, and filter out items they have already seen.

• Semantic search by meaning
• Dedup: similarity above ~0.95
• Recommend: liked item as query

Mistake number one: no embedding model version tracking. Months later nobody knows which model made the vectors, and quality silently drops after a partial upgrade because queries and documents no longer match. Record the model name and version on each record and on the collection, and check it at query time.

• Nobody knows which model
• Quality drops silently
• Tag every record and collection

Mistake number two: over-fetching. Asking for the top one hundred when you display five wastes latency and network, and in RAG it floods the context with noise. Fetch what you need plus a deliberate margin if a reranker follows. And avoid returning embeddings and full payloads when you only need identifiers.

• Top 100 to show 5
• Wasted latency and bandwidth
• Fetch needed plus planned margin

One more capability to watch for is hybrid search, which blends keyword scoring with vector similarity. Many systems support it, because exact terms like product codes and error numbers are where embeddings are weak. We will build it properly, with reciprocal rank fusion, in lesson nineteen.

• Exact terms: embeddings are weak
• Blend keyword and vector scores
• Lesson 19: reciprocal rank fusion

Recap. An index is not a database. A vector database adds persistence, CRUD, metadata, filtering, and scaling. Filtering interacts with approximate search, so measure filtered recall. Use stable ids, tenant isolation and embedding versions. And size memory with arithmetic: dimension times four bytes times count, plus index, metadata, and the number of copies.

• Index is not a database
• Measure filtered recall
• Stable ids, tenant isolation, versions
• Size: dims x 4 x count + overhead

That closes the embeddings section. Next, lesson fifteen: the RAG architecture, where we connect this store to a language model so its answers are grounded in your own data.

• Embeddings section complete
• Connect store to an LLM
• Subscribe for lesson 15