nestonexStart a project What is retrieval-augmented generation, and what problem does it actually solve?
RAG answers a question by first retrieving relevant passages from a collection you control, then asking a language model to answer using those passages. It solves the problem of a model not knowing your material. It does not, on its own, solve accuracy, permissions or the model stating things the passages never said.
Without retrieval, a model answers from what it absorbed during training. It has never seen your contracts, your product documentation or last Tuesday's policy change, and when asked about them it will produce something confident and shaped like an answer.
RAG changes the input, not the model. A query is used to find passages from a collection you control, those passages are placed in the prompt, and the model is asked to answer from them. The knowledge problem becomes a search problem, which is a much better problem to have because search is inspectable.
That is the whole mechanism. Everything that makes RAG work or fail in production is downstream of it: how the collection is split, how passages are found and ranked, what the prompt does when nothing relevant comes back, and whether anyone checks the result.
It does not make a model truthful. A model handed five passages can still combine them into a claim none of them supports, particularly when the passages nearly answer the question. Grounding raises the floor; it does not remove the need to evaluate.
It does not enforce permissions. A retrieval layer that searches every document will happily surface a passage from a document the asker was never allowed to open. Access control belongs in the retrieval query, not in an instruction asking the model to be careful.
It does not fix a corpus. If your documentation contradicts itself across three versions, retrieval returns the contradiction, and the model resolves it by picking one. Retrieval quality is bounded by source quality, and no amount of prompt work moves that ceiling.
And it does not remove the need for a refusal path. The most valuable thing a retrieval assistant can say is that it does not have the answer, which requires deciding in advance what counts as not having it.
Chunking. How the source is split decides what can be retrieved at all. Split too small and a passage loses the context that made it meaningful; too large and the relevant sentence is buried among four paragraphs of noise. Splitting on word or section boundaries with overlap beats splitting on a character count.
Retrieval and ranking. Semantic similarity finds passages that mean something similar; keyword matching finds passages containing the exact term. Each fails in ways the other covers, which is why hybrid ranking usually beats either alone, and why the honest first step is checking whether semantic embeddings actually beat a keyword baseline on your material.
The prompt contract. What the model is told to do when passages conflict, when they are thin, and when they are absent. A prompt that never permits refusal guarantees invention.
The evaluation set. A fixed list of real questions with known-good answers, including the ones that should be refused. Without it, every change to chunking, model or prompt is a guess, and a regression is something a customer reports.
When the corpus is small and stable, putting the whole thing in the prompt is simpler, cheaper to reason about and more accurate than retrieving from it. Context windows have grown enough that this covers more cases than it used to.
When the questions are predictable, a handful of curated answers behind a search box will beat a retrieval assistant on accuracy, latency and cost, and it can be corrected by editing a line of text rather than by retuning a pipeline.
And when the real requirement is a lookup against structured data, retrieval over prose is the wrong shape entirely. A query against the database answers it exactly, every time, and can be tested.
We built a RAG pipeline from first principles as an R&D project, deliberately without a managed service, so that every decision in it stayed visible: word-boundary chunking, one embedder interface hiding both local sentence-transformers and a TF-IDF baseline, L2-normalised vectors, and a hand-rolled in-process store.
The TF-IDF backend is there for a reason. It answers the question most RAG projects skip, which is whether semantic embeddings are beating plain keyword matching on this particular corpus. Often they are. Sometimes they are not, and finding that out costs an afternoon rather than a quarter.
They solve different problems. Fine-tuning changes how a model behaves — its format, tone or task competence — and is expensive to redo when facts change. RAG changes what the model knows at question time and updates the moment you update a document. For company knowledge that changes, retrieval is almost always the right tool.
It reduces it substantially and does not eliminate it. A model can still assert something the retrieved passages do not support, particularly when they nearly answer the question. Citations that a reader can check, a refusal path for thin retrieval, and an evaluation set covering both are what keep it honest.
Not at small to moderate scale. With normalised embeddings, similarity is a dot product over an in-process array, which runs in milliseconds for tens of thousands of chunks. A vector database earns its place when the corpus outgrows memory, needs concurrent writers, or needs filtered search at speed.
By filtering at retrieval, using the asker's identity, before anything reaches the model. Instructing a model not to reveal restricted material is not access control — the passage is already in the prompt by then, and the instruction is one clever question away from being ignored.