A Reference Architecture for Enterprise RAG (That Won’t Hallucinate Your Docs)

Every RAG demo works. You point a model at your docs, ask a question, get a fluent answer with a citation, and the room nods. Then it ships, someone asks about a policy that changed last quarter — and the system confidently cites the old version. The demo never showed you that failure mode, because the failure mode isn’t “the model can’t answer.” It’s “the model answers anyway.”

That gap is where enterprise RAG loses trust — and it’s almost never the model’s fault. It’s retrieval handing over plausible-but-wrong context that a capable model then synthesizes beautifully. The tool is not the transformation: a bigger model makes a confident wrong answer more convincing, not less. The fix lives below the model, in retrieval and grounding. Here’s the architecture I’d stand up for RAG you can actually put in front of employees or customers.

The failure mode: fluent, cited, and wrong

Naive RAG is three lines: embed the query, pull the top-k nearest chunks, stuff them in the prompt. It fails three ways at once. Vector search alone misses the exact terms — part numbers, error codes, policy names — that keyword search would nail. Without reranking, “near in embedding space” isn’t “actually answers the question.” And with no relevance floor, the pipeline always returns something, so the model always has context to rationalize an answer from — even when the right answer is “we don’t have that.”

The result is the worst kind of wrong: confident, well-written, and cited. Everything below exists to make the system able to say “I don’t have that in the sources.”

The reference architecture

  Query
     |
     v
  Hybrid Retrieval      BM25 (keyword) + vector
     |
     v
  Rerank                cross-encoder scores the union
     |
     v
  Relevance floor  --- nothing relevant --->  Refuse ("not in the sources")
     |
  context clears floor
     |
     v
  Grounded generation   answer ONLY from context, cite every claim
     |
     v
  Answer + citations
     |
     v
  Eval + observability  groundedness + citation checks  --->  tune retrieval

1. Ingestion & chunking. How you split documents is a retrieval decision, not a preprocessing chore. Chunk too big and the reranker drowns in noise; too small and you sever the context a claim needs. Preserve structure (headings, tables) and keep the source metadata — you’ll need the doc_id for citations.

2. Hybrid retrieval. Run keyword (BM25) and vector search and take the union. Keyword catches the exact strings vectors miss; vectors catch the meaning keywords miss. Neither alone is enough in the enterprise, where half the questions hinge on a specific identifier.

3. Rerank. Score the union with a cross-encoder that reads the query and each chunk together. This is the highest-ROI line most teams aren’t running — it reorders “nearby” into “actually relevant.”

4. The grounding gate (relevance floor). The seam. If nothing clears a relevance threshold, return nothing. This is where your operating DNA — what the system is allowed to answer, and from what — stops being a hopeful prompt and becomes code. An empty retrieval result is a feature: it’s what lets the generator refuse instead of improvise.

5. Grounded generation. Instruct the model to answer only from the provided context and cite a doc_id for every claim. With no context, it refuses — by construction, not by good behavior.

6. Eval & observability. Score every answer for groundedness and citation validity, continuously. Without this you can’t tell an improvement from a regression — you’re just guessing with a chatbot.

The grounding gate, in code

Hybrid recall, rerank, and a relevance floor — the floor is the part that makes refusal possible:

def retrieve(self, query, k=5):
    # Hybrid recall: keyword catches exact terms, vectors catch meaning.
    candidates = self._dedupe(
        self.bm25.search(query, k=20) + self.vector_index.search(query, k=20)
    )
    # Rerank the union with a cross-encoder — the single biggest quality win.
    for c in candidates:
        c.score = self.reranker.score(query, c.text)
    candidates.sort(key=lambda c: c.score, reverse=True)
    # Relevance floor: if nothing is truly relevant, return nothing.
    # An empty result is a feature — it lets the generator refuse.
    return [c for c in candidates[:k] if c.score >= self.min_score]

And the grounding instruction that turns “no context” into “no answer”:

GROUNDING_INSTRUCTION = (
    "Answer ONLY from the provided context. Cite the doc_id for every claim. "
    "If the context does not contain the answer, say 'I don't have that in the "
    "sources' — do not use prior knowledge."
)

Measuring whether it’s actually grounded

Two numbers predict trust: is every claim supported by a retrieved chunk, and does every cited doc_id actually appear in the context? Score them on a fixed set of questions on every prompt, model, or index change.

def score(case):
    groundedness = case.claims_supported / case.claims_total
    hallucinated = [d for d in case.cited_doc_ids if d not in case.context_doc_ids]
    return {"groundedness": groundedness, "citation_valid": not hallucinated}

def retrieve(self, query, user, k=5):
    candidates = self._dedupe(
        self.bm25.search(query, k=40) + self.vector_index.search(query, k=40)
    )
    # Trim to what THIS user may see — enforced at the data layer, BEFORE rerank,
    # so unauthorized chunks never influence ranking or reach the model at all.
    candidates = [c for c in candidates if can_read(user, c.doc_id)]

    for c in candidates:
        c.score = self.reranker.score(query, c.text)
    candidates.sort(key=lambda c: c.score, reverse=True)
    return [c for c in candidates[:k] if c.score >= self.min_score]

The parts everyone underestimates

Chunking is a retrieval decision. More RAG quality is won or lost here than in model choice. Test chunk sizes against your eval set like any other parameter.

Reranking is the cheap win you’re skipping. A cross-encoder over the top ~40 candidates typically moves groundedness more than a model upgrade — for a fraction of the cost.

The relevance floor is the whole ballgame. “Always answers” is the root of hallucinated RAG. A threshold that returns nothing when nothing is relevant is what converts a plausible liar into a trustworthy assistant.

Retrieval decays silently. A doc gets superseded, and the old chunk still ranks — confidently. Freshness and provenance on your index matter more than the model you pick.

The takeaway

RAG trustworthiness is a retrieval-and-grounding problem, not a model problem. What the system is allowed to answer — and from what — is operating DNA, and it belongs in code at the grounding gate, not in a prompt you hope the model respects. Get that seam right and you can swap models freely as they improve. Skip it and you’ve built a very persuasive way to cite the wrong document.

The retriever and the groundedness eval from this post are runnable on GitHub: github.com/waghmaredb/vexpose-labs. Building enterprise RAG and fighting the same failure modes? I’d like to compare notes — reach me on LinkedIn or X.