Every RAG demo works. You point a model at your docs, ask a question, get a fluent answer with a citation, and the room nods. Then it ships, someone asks about a policy that changed last quarter — and the system confidently cites the old version. The demo never showed you that failure mode, because the failure mode isn’t “the model can’t answer.” It’s “the model answers anyway.”
That gap is where enterprise RAG loses trust — and it’s almost never the model’s fault. It’s retrieval handing over plausible-but-wrong context that a capable model then synthesizes beautifully. The tool is not the transformation: a bigger model makes a confident wrong answer more convincing, not less. The fix lives below the model, in retrieval and grounding. Here’s the architecture I’d stand up for RAG you can actually put in front of employees or customers.
The failure mode: fluent, cited, and wrong
Naive RAG is three lines: embed the query, pull the top-k nearest chunks, stuff them in the prompt. It fails three ways at once. Vector search alone misses the exact terms — part numbers, error codes, policy names — that keyword search would nail. Without reranking, “near in embedding space” isn’t “actually answers the question.” And with no relevance floor, the pipeline always returns something, so the model always has context to rationalize an answer from — even when the right answer is “we don’t have that.”
The result is the worst kind of wrong: confident, well-written, and cited. Everything below exists to make the system able to say “I don’t have that in the sources.”
The reference architecture
Query
|
v
Hybrid Retrieval BM25 (keyword) + vector
|
v
Rerank cross-encoder scores the union
|
v
Relevance floor --- nothing relevant ---> Refuse ("not in the sources")
|
context clears floor
|
v
Grounded generation answer ONLY from context, cite every claim
|
v
Answer + citations
|
v
Eval + observability groundedness + citation checks ---> tune retrieval
1. Ingestion & chunking. How you split documents is a retrieval decision, not a preprocessing chore. Chunk too big and the reranker drowns in noise; too small and you sever the context a claim needs. Preserve structure (headings, tables) and keep the source metadata — you’ll need the doc_id for citations.
2. Hybrid retrieval. Run keyword (BM25) and vector search and take the union. Keyword catches the exact strings vectors miss; vectors catch the meaning keywords miss. Neither alone is enough in the enterprise, where half the questions hinge on a specific identifier.
3. Rerank. Score the union with a cross-encoder that reads the query and each chunk together. This is the highest-ROI line most teams aren’t running — it reorders “nearby” into “actually relevant.”
4. The grounding gate (relevance floor). The seam. If nothing clears a relevance threshold, return nothing. This is where your operating DNA — what the system is allowed to answer, and from what — stops being a hopeful prompt and becomes code. An empty retrieval result is a feature: it’s what lets the generator refuse instead of improvise.
5. Grounded generation. Instruct the model to answer only from the provided context and cite a doc_id for every claim. With no context, it refuses — by construction, not by good behavior.
6. Eval & observability. Score every answer for groundedness and citation validity, continuously. Without this you can’t tell an improvement from a regression — you’re just guessing with a chatbot.
The grounding gate, in code
Hybrid recall, rerank, and a relevance floor — the floor is the part that makes refusal possible:
def retrieve(self, query, k=5):
# Hybrid recall: keyword catches exact terms, vectors catch meaning.
candidates = self._dedupe(
self.bm25.search(query, k=20) + self.vector_index.search(query, k=20)
)
# Rerank the union with a cross-encoder — the single biggest quality win.
for c in candidates:
c.score = self.reranker.score(query, c.text)
candidates.sort(key=lambda c: c.score, reverse=True)
# Relevance floor: if nothing is truly relevant, return nothing.
# An empty result is a feature — it lets the generator refuse.
return [c for c in candidates[:k] if c.score >= self.min_score]
And the grounding instruction that turns “no context” into “no answer”:
GROUNDING_INSTRUCTION = (
"Answer ONLY from the provided context. Cite the doc_id for every claim. "
"If the context does not contain the answer, say 'I don't have that in the "
"sources' — do not use prior knowledge."
)
Measuring whether it’s actually grounded
Two numbers predict trust: is every claim supported by a retrieved chunk, and does every cited doc_id actually appear in the context? Score them on a fixed set of questions on every prompt, model, or index change.
def score(case):
groundedness = case.claims_supported / case.claims_total
hallucinated = [d for d in case.cited_doc_ids if d not in case.context_doc_ids]
return {"groundedness": groundedness, "citation_valid": not hallucinated}
The guardrail everyone forgets: retrieval-time access control
Every architecture above assumes one reader is allowed to see every document. In a real enterprise they are not. HR files, board decks, a customer’s data walled off from another customer’s — the same index holds chunks that different users have different rights to. Miss this and you haven’t built a helpful assistant; you’ve built the most efficient data-exfiltration path in the company. The fastest way to leak the salary spreadsheet is to embed it and let anyone ask.
The critical design decision: access control lives in retrieval, not in the prompt. Filter the candidate set to what this caller is entitled to see before anything is reranked or shown to the model — sourced from the same entitlements as the system of record (row-level security, document ACLs), not re-invented in the pipeline.
def retrieve(self, query, user, k=5):
candidates = self._dedupe(
self.bm25.search(query, k=40) + self.vector_index.search(query, k=40)
)
# Trim to what THIS user may see — enforced at the data layer, BEFORE rerank,
# so unauthorized chunks never influence ranking or reach the model at all.
candidates = [c for c in candidates if can_read(user, c.doc_id)]
for c in candidates:
c.score = self.reranker.score(query, c.text)
candidates.sort(key=lambda c: c.score, reverse=True)
return [c for c in candidates[:k] if c.score >= self.min_score]
A system prompt that says “only answer if the user is authorized” is not a security control — it’s a polite request to a probabilistic system, and it will eventually say yes. Real access control is a hard filter on the retrieval set, applied before rerank so an unauthorized chunk can’t even tilt the ranking. This is the RAG expression of least privilege — the same operating DNA that governs a good agent: the model may ground only on what this specific user could already open for themselves. That’s a decision you enforce in code, not one you hope the model respects.
The parts everyone underestimates
Chunking is a retrieval decision. More RAG quality is won or lost here than in model choice. Test chunk sizes against your eval set like any other parameter.
Reranking is the cheap win you’re skipping. A cross-encoder over the top ~40 candidates typically moves groundedness more than a model upgrade — for a fraction of the cost.
The relevance floor is the whole ballgame. “Always answers” is the root of hallucinated RAG. A threshold that returns nothing when nothing is relevant is what converts a plausible liar into a trustworthy assistant.
Retrieval decays silently. A doc gets superseded, and the old chunk still ranks — confidently. Freshness and provenance on your index matter more than the model you pick.
The takeaway
RAG trustworthiness is a retrieval-and-grounding problem, not a model problem. What the system is allowed to answer — and from what — is operating DNA, and it belongs in code at the grounding gate, not in a prompt you hope the model respects. Get that seam right and you can swap models freely as they improve. Skip it and you’ve built a very persuasive way to cite the wrong document.
The retriever and the groundedness eval from this post are runnable on GitHub: github.com/waghmaredb/vexpose-labs. Building enterprise RAG and fighting the same failure modes? I’d like to compare notes — reach me on LinkedIn or X.
