Document RAG with Measured Retrieval
THE PROBLEM
Reviewers and support staff were drowning in unstructured documents, digging through a shared drive with keyword search that returned files, not answers. The naive RAG I prototyped first looked great on clean demos and fell apart on the real corpus — naive chunking split tables across boundaries and retrieval silently returned plausible garbage. I rebuilt it with retrieval quality measured on a golden set, so regressions get caught by numbers before users catch them.
CONSTRAINTS
- Chunking strategy silently decides retrieval quality — found out on real files
- Vector recall alone misses keyword-heavy queries
- Answers must cite sources or reviewers will not trust them
- Agents need programmatic access, not just a chat UI
DECISIONS & TRADEOFFS
HYBRID SEARCH PLUS RERANKING over vector-only retrieval — because dense embeddings miss exact terms and reranking rescues precision without retraining anything
GOLDEN-SET EVALS over vibes-based tuning — because precision and recall on a fixed question set turn prompt and chunking changes into measurable decisions
MCP SERVER over bespoke API integrations — because exposing retrieval as tools over a standard protocol lets any agent client use it without custom glue code
ARCHITECTURE
OUTCOME
Reviewers get cited answers instead of a folder of search results, and they can check the source text in one click, which is why they trust it. Retrieval quality is a tracked number, not a feeling: chunking and prompt changes are accepted or rejected by the eval harness. The same retrieval layer serves the UI and any MCP-capable agent, so there is one system to keep honest, not two.
PROOF / LINKS
INTERACTIVE DEMO
Document RAG with Measured Retrieval
OPEN ↗
AT 100× SCALE
At 100x this becomes the retrieval backbone for a fleet of agents: per-corpus access control, incremental index updates, and eval gates that block a bad embedding-model swap before users notice.