MZQMohd Zamin Quadri
All work

Retrieval evaluation

InsureAssist: A Measured RAG Benchmark

Answers questions about flood insurance policies, with sources.

Role
Project author and engineer
Classification
Reference implementation
Inspect repository
The nearest passage can be the right provision from the wrong policy form. Three near-duplicate forms make that the whole problem.

  1. Problem

    Near-identical policy forms fool a topic search.

  2. How it was solved

    I tuned chunk size and combined keyword and vector search.

    1. Chunk
    2. Hybrid search
    3. Cite

  3. Result

    0.167 → 0.556

    questions matched to the right form

    • Starting point16.7%
    • Final system55.6%

  4. Built with

    • Qdrant
    • BM25
    • BGE embeddings
    • FastAPI
    • Docker
    See the evidence on GitHub

Reference implementation / Retrieval

Three policy forms differ by a few words. The nearest passage can be the correct clause taken from the wrong document, and read exactly like the answer.

Form discrimination0.556 top-document accuracy

Two retrievers, one fusion rule, and a citation that resolves

embedtokenisedense ranklexical ranktop-k contextsper contextresolves to offsetsNFIP corpusLicensed federalflood-insurance text…Chunk and identifyOffsets kept. IDs derived fromdocument, offset and text, so…Dense indexCosine similarity overembedded chunks.Lexical indexOkapi BM25, written out ratherthan imported.Rank fusionReciprocal rank, k=60.Positions only - the two…Prompt and answerContexts are numbered in theprompt so the answer can refer…CitationsChunk ID, document, CFRcitation and character…Query serviceAnswer and citations returnedtogether, never separately.
Dense and lexical ranks are never averaged - they are on different scales, so reciprocal rank fusion reads only their positions. The path back to the corpus is what makes a citation checkable rather than decorative.

This diagram in words

  1. NFIP corpus source — Licensed federal flood-insurance text, committed to the repository.
  2. Chunk and identify process — Offsets kept. IDs derived from document, offset and text, so re-ingest is idempotent.
  3. Dense index store — Cosine similarity over embedded chunks.
    Qdrant · BGE embeddings
  4. Lexical index store — Okapi BM25, written out rather than imported.
    BM25
  5. Rank fusion gate — Reciprocal rank, k=60. Positions only - the two scales are never added.
  6. Prompt and answer process — Contexts are numbered in the prompt so the answer can refer to them.
    Ollama
  7. Citations artifact — Chunk ID, document, CFR citation and character offsets. Never generated.
  8. Query service sink — Answer and citations returned together, never separately.
    FastAPI

01 / 11Three formsThe NFIP Standard Flood Insurance Policy, as three federal forms. Redistributable: works of the US Government.

  1. Dwelling Form44 CFR Part 61, Appendix A(1) to Part 61 · 12,603 words
  2. General Property Form44 CFR Part 61, Appendix A(2) to Part 61 · 11,262 words
  3. RCBAP44 CFR Part 61, Appendix A(3) to Part 61 · 11,774 words
hit@5top-document accuracyDense only (BGE)0.50.5556BM25 only0.61110.3333Hybrid RRF0.55560.5556
Measured on 3 forms and 314 chunks, same labels and same held-out split. BM25 alone retrieves the most relevant chunks and is the worst at returning the right form; the selected hybrid is second on one and joint-first on the other. 8 of 18 held-out questions still put the wrong form at rank one, down from a configuration that got the form right 16.7% of the time.

The corpus is the three forms below, published by the Federal Emergency Management Agency in the Code of Federal Regulations. They are redistributable under 17 U.S.C. 105: works of the United States Government are not subject to copyright protection.

  • Dwelling Form44 CFR Part 61, Appendix A(1) to Part 61
  • General Property Form44 CFR Part 61, Appendix A(2) to Part 61
  • RCBAP44 CFR Part 61, Appendix A(3) to Part 61
  • Twenty-two held-out questionsCategory cells of two to five carry no reliable signal, and per-form numbers are not reported as findings.
  • Dev did not generaliseHit@5 was 1.0 on dev and 0.5556 on test - the honest cost of selecting on 14 answerable questions.
  • It never abstainsUnanswerable rejection rate is 0.0. Every unanswerable question is answered anyway, and no threshold is defensible on this data.
  • One jurisdiction, one perilThree US flood policy forms. The result describes this wording, not insurance documents generally.

Held-out MRR 0.42, hit@5 0.5556, top-document accuracy 0.5556. Every figure here is read from the repository’s frozen reference run.

Technical questionHow was the chunk size chosen, and does the score that chose it survive the held-out set?

1.00

hit@5 on the 14 questions it was chosen on

0.5556

hit@5 on the 18 it was not

Selected on development questions, reported on held-out ones. The first number is what selection produces; the second is the one that means anything.

Development sweep over 14 answerable questions. The frozen configuration is 800 characters with 120 of overlap, giving 314 chunks. Held-out hit@5 is 0.5556. Source: eval/dev_chunking_sweep.json
Chunk sizeOverlapDepthChunkshit@5MRR
40080106640.42860.3824
40080206640.50000.4167
600100104260.64290.4923
600100204260.78570.5126
600150104740.71430.5185
600150204740.64290.5211
800120103141.00000.6214
800120203140.92860.6071
1000150102510.71430.6015
1000150202510.71430.6066
Read the full background

Problem

Why this work exists

The three NFIP policy forms are near-duplicates by design: they share a skeleton and much verbatim wording but differ in substance. A retriever matching on topic finds the right provision in the wrong document, and nothing in a demo reveals that.

Contribution

What I can claim

Built a licensed corpus with hash-verified provenance, deterministic ingestion with content-derived chunk IDs, a 40-question labelled benchmark with a held-out split, a lexical baseline to compare against, and a machine-readable reference run that the repository's own CI checks its documentation against.

Method

Why retrieval quality and answer quality are two different results

A retrieval-augmented system fails in two distinct ways, and reporting one number hides both.

  1. Ingest

    Source documents are split into passages.

  2. Represent

    Each passage takes a position in a space where distance is meant to stand for relevance.

  3. Retrieve

    A question arrives and the nearest passages are selected.

  4. Ground

    The answer is tied back to the passages it was built from.

  5. Verify

    Retrieval is scored separately from generation, so a fluent answer built on the wrong passage is visible as the failure it is.

  6. What this does not show

    Retrieval scores in the evidence table are measured on a fixed question set over public policy text. They say nothing about performance on other corpora, other question styles, or questions whose answer is not present at all.

How it was built

System

Workflow and decisions

Retrieval is dense BGE vectors fused with an in-process BM25 index by reciprocal rank fusion. The architecture was selected on a development split and frozen before the held-out split was run once.

  1. 1Licensed NFIP corpus
  2. 2Deterministic chunking and IDs
  3. 3Dense and lexical retrieval
  4. 4Rank fusion
  5. 5Traceable citations
  • FastAPI
  • Qdrant
  • BGE embeddings
  • BM25
  • Ollama
  • Docker
  • Kubernetes
  • GitHub Actions

Evidence

What is actually versioned

Form discrimination0.556 top-document accuracy

Up from 0.167 at the starting point: the share of questions whose best hit comes from the correct policy form, on the held-out split.

Labelled benchmark50 questions

32 answerable and 18 unanswerable, each naming exact chunk IDs and character offsets across 314 chunks.

Citation integrity0 unsupported

Every citation's offsets reproduce its quoted text from the committed corpus, so no provenance is fabricated.

Inspection points

Go directly to the evidence

Benchmark method and resultsHeld-out metrics, baselines, failure analysis, and what the numbers do not say.Reference runMachine-readable artefact every published number derives from.Frozen retrieval configThe architecture and parameters, with the dev evidence that selected them.Corpus provenanceSource, legal basis, extraction method, and per-document hashes.LimitationsThe complete list, including the results that did not improve.
How the work is checked

Quality controls

How the work is checked

  • The retrieval architecture was chosen on a development split and frozen in a committed config before the held-out split was run once
  • Chunk IDs derive from document, offset and text, so a relevance label still points at its evidence after re-ingestion
  • Ground-truth labels resolve from text anchors, so a chunking change regenerates them instead of invalidating the benchmark
  • A validator rejects broken references, spans outside a document, and unanswerable questions carrying evidence
  • 447 offline tests run with no model, no database and no network
  • CI reproduces the reference run against a real vector database and fails if the repository's documentation drifts from it

Limitations

Where the evidence stops

  • The lexical baseline alone still retrieves more relevant chunks in the top five than the selected hybrid; the hybrid was chosen for ranking and form discrimination, not raw recall
  • Development results did not generalise: the selected architecture scored far higher on the split it was chosen on than on the held-out split
  • Unanswerable questions are not detected. No similarity threshold was defensible on this data, so none is claimed
  • Answer quality is not measured; only one local model is available and grading its own output would be circular
  • One jurisdiction, one peril, three documents. Results do not transfer to insurance documents generally
  • The Kubernetes manifests have been applied only to a local kind cluster, and the Azure Container Apps deployment hosts the API alone, with its readiness probe reporting the vector store absent by design; no full stack runs in the cloud, and the Google Cloud guide has never been run
  • A fine-tuning notebook exists but is archived: its training data was the earlier evaluation set, so no tuned-model result could be honest
What this changed in my practice

What this changed in my practice

A benchmark is only worth what its hardest cases are worth. Near-duplicate documents exposed a failure that topic-level retrieval metrics would have hidden entirely.
Source and documentationDiscuss relevant work

LearnThe reasoning, written up

tutorial5 min read

Hybrid Retrieval Without Silent Failure

Dense and sparse retrieval fused server-side, embedding contracts that are checked rather than assumed, and refusing to index rather than storing a truncated vector.

Retrieval and Grounded Generation
tutorial5 min read

Production AI Systems Need Refusal Paths

Every stage that can degrade should be able to decline. Where refusal belongs across ingestion, retrieval, extraction and inference, and what omitting it costs.

Reliable AI Systems
tutorial5 min read

Template-Driven Document Extraction With LLMs

Schema-validated extraction with a deterministic fallback, corruption detection before the model sees the text, and a contract that survives replacing the model.

Document Intelligence