RAG without a vector database: when a simpler store is enough

Why a plain array is often enough

Cosine similarity divides the dot product of two vectors by the product of their magnitudes. Normalise every vector to unit length when you store it and both magnitudes are one, so the division does nothing and similarity collapses to a dot product.

A dot product against every stored vector is a single matrix multiplication. For tens of thousands of chunks that runs in milliseconds in-process, with no service to deploy, no index to tune, and no network hop between your code and your data.

That is not a compromise for a small corpus. It is exact rather than approximate, one dependency lighter, and considerably easier to reason about when retrieval starts returning something odd.

• Normalise at write time so similarity is a dot product
• An exact scan of tens of thousands of chunks is milliseconds
• No index tuning, no extra service, no network hop

Where Postgres fits

If you already run Postgres, storing embeddings in it is usually the right middle step. You get persistence, transactions, backups and the ability to filter on ordinary columns in the same query — the last of which is the thing hand-rolled stores handle badly.

Filtered search is the requirement that catches teams out. Restricting retrieval to one customer's documents, one date range or one document type is trivial in SQL and awkward when your store is an array you scan linearly. If your product is multi-tenant, you need this on day one rather than at scale.

Extensions add approximate indexing when exact scans stop being fast enough, which means the migration from exact to approximate happens inside a database you already operate rather than as a new service with its own failure modes.

The thresholds that actually force the decision

Memory comes first. Vectors are floats, and a few hundred thousand chunks at typical dimensions is real memory you probably do not want resident in an application process.

Concurrency comes second. An in-process store belongs to one process. The moment you need several writers, or several machines reading a shared index, the store has to move outside the process.

Latency under filtered load comes third, and it usually arrives before raw size does. An exact scan is linear, so it degrades predictably — which is fine until a filtered query has to scan everything before filtering.

Notice that none of these thresholds is about answer quality. They are about scale and operations, which is exactly why reaching for a vector database to fix bad answers does not work.

What actually decides retrieval quality

Chunking. How the source is split decides what can be retrieved at all. Splitting on word and section boundaries with overlap keeps a fact that straddles a boundary retrievable; splitting on a character count guarantees that some of them are not.

Ranking. Semantic similarity finds passages that mean something similar; keyword matching finds passages containing the exact term. Identifiers, negation and rare domain terms defeat the first and are trivial for the second, which is why hybrid ranking usually wins.

And the baseline nobody builds. Implement TF-IDF keyword retrieval alongside the embeddings and compare on your own corpus. If keyword matching is within a few points, the embeddings are buying complexity rather than relevance — and finding that out costs an afternoon.

What we did, and what we would do again

We built a RAG pipeline from first principles as an R&D project, deliberately without a managed service, so every decision stayed visible: word-boundary chunking, one embedder interface hiding both local sentence-transformers and a TF-IDF baseline, normalised vectors, and a hand-rolled in-process store. It runs entirely offline.

The TF-IDF backend is the part we would insist on again. It answers the question most retrieval projects skip, which is whether semantic embeddings are actually beating keyword matching here.

For a client product we would usually start with embeddings in Postgres rather than in memory, because filtered search and persistence arrive early in any real product. The reasoning is the same: pick the simplest store that meets the constraints you can name, and move when a constraint actually binds.