APFlow
Book a call
←  Case studies Financial services

A RAG system that answers policy questions with the paragraph it came from

An operations team of about 400 people answered customer questions by searching a 3,000-document policy library that keyword search could not navigate. A first attempt at a chatbot had been abandoned after it confidently invented a refund rule. We rebuilt it as a retrieval system with citations, an evaluation suite, and a refusal path, and it went from a demo nobody trusted to a tool the team opens by default.

Scope
Retrieval and answering over about 3,000 policy documents
Timeline
8 weeks including the evaluation build
Delivery
Deployed in the client's VPC, source in their repo
Volume
About 1,900 questions a week from around 400 staff
8 minutes → 40 seconds Median time to find a defensible answer to a policy question

01The mess before

The answer existed and nobody could find it

Policies lived across a wiki, a document management system, and a set of PDF circulars going back years. Keyword search returned the same twelve popular documents regardless of the question, so staff asked a colleague instead, and the same question got answered slightly differently depending on who was asked.

A previous chatbot had invented a rule

An earlier attempt produced a fluent, specific, and entirely fictional refund window. It was caught in testing, but it ended the project. The bar for the rebuild was not fluency, it was that every answer had to be traceable to a paragraph a supervisor could open.

Superseded policies looked exactly like current ones

The library contained multiple versions of the same policy with no consistent marking of which was in force. Both a human and a naive retrieval system would happily surface a circular that had been replaced two years earlier.

02What we wired together

Qdrant
Vector store for chunk embeddings, with metadata filters for policy status, effective date, and business line.
PostgreSQL
Document registry, version lineage, full-text index for the keyword half of hybrid search, and the question log.
Embedding and reranking models
A retrieval embedding model for recall, then a cross-encoder reranker to put the genuinely relevant chunk first.
Python retrieval service
Hybrid search, reranking, prompt assembly, citation extraction, and the refusal path when evidence is thin.
Slack
Where most questions are actually asked, so the copilot answers in the channel with its citations attached.
GitHub Actions
Runs the evaluation suite on every change, so a retrieval tweak that improves one question class and breaks another is caught before release.

03How the build ran

  1. Built the evaluation set before the system

    We collected around 300 real questions with answers verified by the policy team, including questions that should be refused because no policy covers them. Nothing shipped until it could be measured against that set. This is the step the abandoned attempt skipped.

  2. Fixed the corpus before touching retrieval

    Deduplicating versions, marking superseded documents, and recording effective dates improved answer quality more than any model change that followed. A retrieval system inherits whatever mess is in the library, and this library had two years of quiet contradictions in it.

  3. Chunked along the document's own structure

    Policy documents have clauses and numbered sections, and cutting on a fixed token count split rules from their exceptions, which is the fastest way to generate a confidently wrong answer. Chunking on structural boundaries, with the section heading carried into each chunk, was worth more than any embedding upgrade.

  4. Made refusal a first-class outcome

    When retrieval scores fall below a threshold the system says it cannot find a policy covering the question and offers the closest documents, rather than assembling an answer from weak evidence. Roughly 8 percent of questions get this response, and it is a large part of why the team trusts the other 92 percent.

  5. Rolled out to one team, then widened

    The first four weeks ran with a single team of about thirty, with a thumbs up or down on every answer. Those ratings drove three rounds of retrieval tuning and added around 60 questions to the evaluation set before anyone else saw the system.

04Under the hood

Hybrid retrieval, because policy questions carry exact terms

Pure vector search missed questions containing specific product codes and circular numbers, where the exact string matters more than the semantics. Combining vector search with full-text search and fusing the results lifted retrieval recall at rank 5 from about 71 percent to about 94 percent on the evaluation set, which was the single largest quality gain in the project.

Reranking is what makes the citation right

Recall gets the right chunk into the top twenty. A cross-encoder reranker is what gets it to position one, and position one is what the answer and its citation are built from. Adding reranking moved top-1 accuracy from about 68 percent to about 89 percent for a cost of roughly 200 milliseconds per query.

Effective dates filter before the model ever sees a chunk

Retrieval filters on policy status and effective date as hard metadata constraints, so a superseded circular cannot reach the answer no matter how semantically similar it looks. This was the specific failure mode that killed the previous attempt, and it is handled in the query rather than in a prompt instruction.

Every claim carries the paragraph behind it

Answers are assembled so each statement links to the chunk it came from, and the interface shows the source paragraph inline rather than a document-level link. Supervisors verify in seconds instead of opening a 40-page PDF, which is the difference between a tool people check and a tool people use.

05Before and after

Metric Before After
Median time to a defensible answer About 8 minutes About 40 seconds
Retrieval recall at rank 5 About 71 percent, vector only About 94 percent, hybrid with reranking
Answers citing a superseded policy The failure that ended the first attempt None in evaluation since date filtering
Questions escalated to the policy team About 240 a week About 55 a week
Where it landed

The policy library went from an archive people avoided to something they query dozens of times a day. Answers arrive in Slack with the paragraph attached, the system says so when no policy covers the question, and the policy team went from answering the same questions repeatedly to working on the gaps the refusal log keeps pointing at.

Have a build like this one?

Book a scoping call →