The difference between RAG that demos well and RAG that survives real questions is what happens when retrieval comes back thin. A basic pipeline generates anyway. An agentic one notices and goes back.
The pipeline, stage by stage
01RewriteResolve vague pronouns and expand domain synonyms into several sub-queries. Cheap, and it fixes a large share of retrieval misses.
02RetrieveHybrid dense plus keyword search, unioned and deduped by document section. Neither method alone covers both paraphrase and exact-term matching.
03RerankA cross-encoder scores each candidate against the original question. Embedding similarity finds topically close passages; the reranker finds correct ones.
04VerifyCheck whether the kept context actually covers the question. A gap triggers one more targeted query rather than a confident guess.
05AnswerEvery claim carries a citation, or it does not get written.
Evaluate retrieval separately from generation
The single most useful habit in this area: score the two halves independently. A bad answer from good context is a generation problem; a good answer from bad context got lucky. Faithfulness, context recall, context precision and answer relevance each tell you which half to fix.
Track unanswerable questions too. A system that confidently answers something its corpus does not cover is worse than one that says it does not know, and only a dedicated eval slice catches it.
