nestonexStart a project Do you actually need a vector database to build retrieval-augmented generation?
Not at small to moderate scale. If embeddings are L2-normalised, similarity is a dot product and retrieval is one matrix operation over an in-process array. A dedicated vector database earns its place when the corpus outgrows memory, needs concurrent writers, or requires filtered search at speed.
Cosine similarity divides the dot product of two vectors by the product of their magnitudes. Normalise every vector to unit length when you store it and both magnitudes are 1, so the division does nothing and similarity reduces to a dot product.
A dot product across every stored vector is one matrix multiplication. For tens of thousands of chunks that runs in milliseconds on a laptop, and there is no service to deploy, no index to tune and no network hop between your code and your data.
Memory is the first wall. Vectors are floats, and a few hundred thousand chunks at typical dimensions will exhaust what you want to hold in a process.
Concurrency is the second. An in-process store belongs to one process, so the moment you need several writers or several machines reading a shared index, you need something outside the process.
Filtered search is the third and the one people hit unexpectedly. Restricting a search to one customer's documents, or one date range, is trivial in a database and awkward when your store is an array you scan linearly.
Approximate nearest neighbour indexing, mostly. A vector database uses structures like HNSW to avoid comparing against every vector, which is what keeps queries fast as the corpus grows. A linear scan is exact but scales linearly.
For a corpus where a full scan takes milliseconds, exact and simple beats approximate and operationally heavier. That calculus flips somewhere between hundreds of thousands and millions of chunks, depending on your latency budget.
Not the store. Chunking and ranking decide whether the right passage comes back, and both are independent of where the vectors live.
Splitting on word boundaries rather than character counts, using overlap so a fact straddling a boundary stays retrievable, and keeping source and offset alongside each vector so an answer can be traced. Teams reach for a vector database hoping it fixes retrieval, and it does not, because the store was never the problem.
Our RAG pipeline was built from first principles as an R&D project, deliberately without a managed service, to make every decision visible and adjustable.
One embedder interface hides two backends, local sentence-transformers and a TF-IDF baseline, so you can check whether semantic embeddings are actually beating keyword matching on your corpus. The whole thing runs offline with no external service at all.
There is no fixed number, but the practical trigger is when vectors stop comfortably fitting in memory or when you need concurrent writers. Tens of thousands of chunks are fine in process. Millions are not. Filtered search at scale usually forces the decision before raw size does.
Only when both vectors are normalised to unit length. Cosine divides the dot product by both magnitudes, so once those are 1 the division is a no-op. Normalising at write time is what lets you use the cheaper operation on every query afterwards.
A vector store is anywhere embeddings live and can be searched, including an array in your process. A vector database is a managed service adding ANN indexing, persistence, filtering and concurrent access. The database is one implementation of the store, not a synonym for it.
Rarely on its own. It improves speed and scale, not relevance. Answer quality is governed by chunking, what you embed and how you rank results. If retrieval is returning the wrong passages, changing where the vectors are stored will not fix it.