Self-Hosted RAG in Production: The Issues That Never Show Up in the Demo

The demo lied to you, not on purpose

Every RAG demo I’ve seen looks great, because it’s tested on the questions its builder already knows the answer to. That’s not dishonest, it’s just how demos get built. Then it ships, real users ask real questions, and it turns out the demo was measuring nothing. Retrieval accuracy was a vibe, hallucination rate was “seemed fine when I tried it,” and cost projections were a spreadsheet built on the assumption that traffic looks like the demo. None of that survives contact with production.

The good news is that every one of these gaps has a concrete, unglamorous fix. None of them require a better model.

Problem 1: you don’t actually know if retrieval works

This is the root cause behind most of what looks like “the model is dumb.” If retrieval fetches the wrong passage, the model is doing exactly what it’s supposed to do: answering confidently based on the context it was handed. The context was just wrong.

You cannot fix what you don’t measure. Build an eval set before you build anything else: 50-100 real questions with known-correct source passages, ideally pulled from actual user queries once you have any, or written by someone who knows the domain if you don’t. Score retrieval with something like:

def retrieval_hit_rate(eval_set, retriever, k=5):
    hits = 0
    for item in eval_set:
        retrieved = retriever.search(item["query"], top_k=k)
        retrieved_ids = {r.doc_id for r in retrieved}
        if item["correct_doc_id"] in retrieved_ids:
            hits += 1
    return hits / len(eval_set)

Run this before and after every change, chunking strategy, embedding model, re-ranker, so “we improved retrieval” is a number that moved, not a feeling. In practice, most retrieval quality problems trace to one of a few specific causes:

• Chunking that ignores document structure. Splitting mid-sentence or mid-table destroys the semantic unit the embedding is supposed to capture. Chunk along headers, paragraphs, or logical sections, not a fixed character count.
• Pure vector search on content full of exact tokens. Error codes, SKUs, product names, and ticket IDs get fuzzed over by semantic similarity. Hybrid search, combining keyword (BM25) with vector search, then a re-ranking pass, consistently outperforms vector-only on real internal documents, because it catches both the natural-language question and the exact-string lookup.
• No re-ranker. Initial retrieval (vector or hybrid) optimizes for recall, pulling in a wider net of plausible candidates. A cross-encoder re-ranker on the top 20-50 candidates before picking the final top-k meaningfully improves precision for not much added latency.

Problem 2: confident hallucination on the questions retrieval can’t answer

Once retrieval is measured and reasonably good, there’s still a class of query it will never nail: out-of-scope questions, ambiguous phrasing, or genuine gaps in the document set. The failure mode here isn’t bad retrieval, it’s a model that doesn’t know how to say “I don’t know” and instead generates something fluent and wrong.

Two guardrails handle most of this:

Grounding checks. Before returning an answer, verify it’s actually supported by the retrieved passages, not just plausible given them. A cheap version is asking the model itself to check its answer against the context as a separate call; a more rigorous version uses a natural language inference model to score entailment between the answer and the source passages, flagging low-entailment answers for a fallback response instead of returning them.

An explicit low-confidence path. If the top retrieved passage’s similarity score falls below a threshold you’ve calibrated against your eval set, don’t let the model try anyway:

results = retriever.search(query, top_k=5)
if not results or results[0].score < CONFIDENCE_THRESHOLD:
    return "I couldn't find anything in the docs that answers this confidently. Here's the closest match I found, but verify it: ..."

This is a worse user experience than a confident wrong answer for exactly one query. It’s a dramatically better experience the first time someone would have acted on a hallucinated policy or a fabricated error code, and it’s what keeps people trusting the system past the first bad answer. Internal RAG in particular lives or dies on this: the first time it confidently quotes a rescinded policy, people go back to asking a colleague and don’t come back.

Problem 3: GPU serving cost doesn’t scale the way API cost does

Self-hosting looks cheaper per token on paper and often is, at the right volume. But the cost model is fundamentally different from an API, and teams that don’t account for the difference get surprised. An API charges you per token you actually use. A GPU you’re renting charges you for every second it’s running, whether or not it’s serving a request. Underutilized GPU capacity is the single biggest way a “we’re saving money by self-hosting” project quietly becomes more expensive than the API it replaced.

The fix is autoscaling GPU capacity to demand rather than running a fixed fleet sized for peak, and being honest about your actual traffic shape before committing to self-hosting at all:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
    - type: prometheus
      metadata:
        query: sum(rate(vllm_request_queue_length[2m]))
        threshold: "10"

And don’t treat self-hosted vs. API as binary. The pattern that’s worked best in practice is a gateway in front of both: route easy, high-volume, low-complexity queries to a self-hosted open model on GPU (served efficiently with vLLM’s continuous batching and PagedAttention), reserve a frontier API for genuinely hard queries, and cache repeated questions so they don’t hit either backend twice. On a self-hosted RAG assistant I built an LLM gateway for, that routing plus caching cut inference cost roughly 70% versus sending everything to a frontier API, with no quality drop, because the easy traffic never needed the expensive model in the first place. The gateway also gives you per-request cost visibility, which matters, because “our LLM bill is $40k this month” is a much harder problem to fix than “query type X costs $0.02 and runs 2M times a month.”

Problem 4: the assistant confidently quotes last year’s policy

Documents change. A RAG system with a one-time ingestion and no re-index strategy will happily keep serving the old version of a policy, price, or procedure indefinitely, with the same confident tone as if it were current. This is the staleness problem, and it’s invisible until someone acts on outdated information and it causes real damage.

The fix scales with how fast your content actually changes. For most internal knowledge bases, a nightly re-index job plus a staleness flag on any source document past some age threshold is enough:

if (datetime.now() - doc.last_verified).days > STALENESS_THRESHOLD_DAYS:
    doc.metadata["stale"] = True

Surface that flag in the answer (“this may be outdated, last verified 94 days ago”) rather than silently trusting the model to know. For faster-moving content, trigger re-indexing directly off the document write event instead of waiting for a nightly batch. On an internal RAG assistant over 12,000 documents, this nightly re-index plus staleness flag was a small addition on top of the retrieval pipeline, and it’s the piece that kept the assistant trusted months after launch instead of just on day one.

The pattern across all four

Every one of these problems has the same shape: something that’s invisible in a demo becomes load-bearing in production. Retrieval quality, hallucination rate, cost per query, and content freshness are all things a demo never forces you to measure, because a demo runs a handful of times on questions someone already knows the answer to. Production runs it thousands of times on questions nobody anticipated, against documents that keep changing, at a cost that compounds. Measuring these four things before you ship, not after users find the gaps, is most of what separates a RAG system that survives contact with real usage from one that gets quietly turned off three months in.

If you’ve got a RAG demo that can’t seem to cross the line into something you’d trust in production, that gap is exactly what RAG systems and AI chatbots work closes, and it’s rarely the model’s fault.