RAG without the hallucinations

When a retrieval system makes something up, the instinct is to blame the model. Usually the model is the only part behaving reasonably. It was handed three chunks of loosely related text and asked to sound confident, and it did exactly that.

Almost every hallucination we are called in to fix turns out to be a retrieval failure wearing a generation costume. The fix is upstream.

Retrieval quality is the whole product

A generous way to test this: take the questions your system got wrong, retrieve for them, and read what came back before the model saw it. If you cannot answer the question from those chunks, no model can, and no amount of prompt tuning will save it. We do this exercise first on every engagement, and it usually ends the debate about which model to use.

If a careful human could not answer from the retrieved context, the model is not hallucinating. It is guessing, because you asked it to.

What actually moves the number is unglamorous: chunk on document structure rather than a fixed character count, keep headings attached to the text beneath them, store enough metadata to filter before you rank, and run hybrid search so exact terms — error codes, product SKUs, surnames — are not smoothed away by an embedding.

Make the model cite, then check the citation

We require answers to quote their sources, and then we verify the quote actually appears in the retrieved text. It is a cheap, mechanical check, and it catches the most damaging failure mode there is: a fluent answer attached to a real-looking citation that says nothing of the kind.

const Answer = z.object({
  answer: z.string(),
  citations: z.array(z.object({
    chunkId: z.string(),
    quote: z.string(),   // must appear verbatim in that chunk
  })).min(1),
  refused: z.boolean(),
})

// Fails closed: unverifiable quote -> refusal, not a confident answer.
if (!quotesAppearInContext(parsed, chunks)) return REFUSAL

Give it somewhere to say no

A system with no refusal path will always answer, because answering is the only move available. So we build the refusal explicitly, reward it in evaluation, and show it in the interface as a normal outcome rather than an error. "I don't have that in the documentation" is a correct response, and users trust a product that can produce it far more than one that never does.

Evals, or you are just vibing

You cannot improve what you only assess by feel. Before launch we build a set of graded questions drawn from real usage — including the awkward ones: questions the corpus genuinely cannot answer, questions with two plausible answers, questions where the right answer changed last quarter.

• Retrieval: is the needed chunk in the top-k at all? Fix this before touching anything else.
• Groundedness: is every claim traceable to retrieved text?
• Refusal accuracy: does it decline when it should, and only then?
• Regression: the suite runs in CI, so a prompt tweak that quietly breaks nine other cases fails the build.

That last one is the difference between a feature and a demo. Prompts are code with unusually poor type-safety; they deserve the same gate as everything else in the repo, for the same reasons set out in performance budgets that actually hold.

Boring infrastructure, on purpose

Most products do not need a dedicated vector database. If your data already lives in Postgres, pgvector keeps retrieval next to the rows it describes, inside the same transaction and the same backup — which matters more in year two than query latency does in week one. We default to boring databases for exactly this reason, and we keep the pipeline model-agnostic so swapping providers is a config change rather than a rewrite.

None of this is exotic. Retrieve honestly, cite verifiably, refuse when you should, measure continuously. The result is a feature people come to rely on instead of one they learn to double-check.