Retrieval evaluation

InsureAssist: A Measured RAG Benchmark

A retrieval-augmented question-answering service over real federal flood-insurance policy text, built so its retrieval quality can be measured rather than demonstrated.

Role: Project author and engineer
Classification: Reference implementation

Inspect repository

The nearest passage can be the right provision from the wrong policy form. Three near-duplicate forms make that the whole problem.

01 / 11Three formsThe NFIP Standard Flood Insurance Policy, as three federal forms. Redistributable: works of the US Government.

Dwelling Form44 CFR Part 61, Appendix A(1) to Part 61 · 12,603 words
General Property Form44 CFR Part 61, Appendix A(2) to Part 61 · 11,262 words
RCBAP44 CFR Part 61, Appendix A(3) to Part 61 · 11,774 words

Measured on 3 forms and 314 chunks, same labels and same held-out split. BM25 alone retrieves the most relevant chunks and is the worst at returning the right form; the selected hybrid is second on one and joint-first on the other. 8 of 18 held-out questions still put the wrong form at rank one, down from a configuration that got the form right 16.7% of the time.

The corpus is the three forms below, published by the Federal Emergency Management Agency in the Code of Federal Regulations. They are redistributable under 17 U.S.C. 105: works of the United States Government are not subject to copyright protection.

Dwelling Form44 CFR Part 61, Appendix A(1) to Part 61
General Property Form44 CFR Part 61, Appendix A(2) to Part 61
RCBAP44 CFR Part 61, Appendix A(3) to Part 61

Twenty-two held-out questionsCategory cells of two to five carry no reliable signal, and per-form numbers are not reported as findings.
Dev did not generaliseHit@5 was 1.0 on dev and 0.5556 on test - the honest cost of selecting on 14 answerable questions.
It never abstainsUnanswerable rejection rate is 0.0. Every unanswerable question is answered anyway, and no threshold is defensible on this data.
One jurisdiction, one perilThree US flood policy forms. The result describes this wording, not insurance documents generally.

Held-out MRR 0.42, hit@5 0.5556, top-document accuracy 0.5556. Every figure here is read from the repository’s frozen reference run.

Technical questionHow was the chunk size chosen, and does the score that chose it survive the held-out set?

1.00

hit@5 on the 14 questions it was chosen on

0.5556

hit@5 on the 18 it was not

Selected on development questions, reported on held-out ones. The first number is what selection produces; the second is the one that means anything.

Development sweep over 14 answerable questions. The frozen configuration is 800 characters with 120 of overlap, giving 314 chunks. Held-out hit@5 is 0.5556. Source: eval/dev_chunking_sweep.json
Chunk size	Overlap	Depth	Chunks	hit@5	MRR
400	80	10	664	0.4286	0.3824
400	80	20	664	0.5000	0.4167
600	100	10	426	0.6429	0.4923
600	100	20	426	0.7857	0.5126
600	150	10	474	0.7143	0.5185
600	150	20	474	0.6429	0.5211
800	120	10	314	1.0000	0.6214
800	120	20	314	0.9286	0.6071
1000	150	10	251	0.7143	0.6015
1000	150	20	251	0.7143	0.6066

01Problem

Why this work exists

The three NFIP policy forms are near-duplicates by design: they share a skeleton and much verbatim wording but differ in substance. A retriever matching on topic finds the right provision in the wrong document, and nothing in a demo reveals that.

02Contribution

What I can claim

Built a licensed corpus with hash-verified provenance, deterministic ingestion with content-derived chunk IDs, a 40-question labelled benchmark with a held-out split, a lexical baseline to compare against, and a machine-readable reference run that the repository's own CI checks its documentation against.

Method

Why retrieval quality and answer quality are two different results

A retrieval-augmented system fails in two distinct ways, and reporting one number hides both. This separates them: what was found, and what was written from it.

Ingest
Source documents are split into passages. The split is part of the system, not preparation for it - a passage boundary in the wrong place removes an answer from reach before retrieval ever runs.
Represent
Each passage takes a position in a space where distance is meant to stand for relevance. Meant to: whether it does is the thing being measured.
Retrieve
A question arrives and the nearest passages are selected. This step can be scored on its own, against known-correct sources, without generating anything.
Ground
The answer is tied back to the passages it was built from. An answer that cannot be traced to its evidence is not a better answer, it is an unverifiable one.
Verify
Retrieval is scored separately from generation, so a fluent answer built on the wrong passage is visible as the failure it is.
What this does not show
Retrieval scores in the evidence table are measured on a fixed question set over public policy text. They say nothing about performance on other corpora, other question styles, or questions whose answer is not present at all.

03System

Workflow and decisions

Retrieval is dense BGE vectors fused with an in-process BM25 index by reciprocal rank fusion. The architecture was selected on a development split and frozen before the held-out split was run once.

01Licensed NFIP corpus
02Deterministic chunking and IDs
03Dense and lexical retrieval
04Rank fusion
05Traceable citations

FastAPI
Qdrant
BGE embeddings
BM25
Ollama
Docker
Kubernetes
GitHub Actions

04Evidence

What is actually versioned

Form discrimination0.556 top-document accuracy

Up from 0.167 at the starting point: the share of questions whose best hit comes from the correct policy form, on the held-out split.

Labelled benchmark40 questions

32 answerable and 8 unanswerable, each naming exact chunk IDs and character offsets across 314 chunks.

Citation integrity0 unsupported

Every citation's offsets reproduce its quoted text from the committed corpus, so no provenance is fabricated.

05Inspection points

Go directly to the evidence

Benchmark method and resultsHeld-out metrics, baselines, failure analysis, and what the numbers do not say.Reference runMachine-readable artefact every published number derives from.Frozen retrieval configThe architecture and parameters, with the dev evidence that selected them.Corpus provenanceSource, legal basis, extraction method, and per-document hashes.LimitationsThe complete list, including the results that did not improve.

06Quality controls

How the work is checked

The retrieval architecture was chosen on a development split and frozen in a committed config before the held-out split was run once
Chunk IDs derive from document, offset and text, so a relevance label still points at its evidence after re-ingestion
Ground-truth labels resolve from text anchors, so a chunking change regenerates them instead of invalidating the benchmark
A validator rejects broken references, spans outside a document, and unanswerable questions carrying evidence
224 offline tests run with no model, no database and no network
CI reproduces the reference run against a real vector database and fails if the repository's documentation drifts from it

07Limitations

Where the evidence stops

The lexical baseline alone still retrieves more relevant chunks in the top five than the selected hybrid; the hybrid was chosen for ranking and form discrimination, not raw recall
Development results did not generalise: the selected architecture scored far higher on the split it was chosen on than on the held-out split
Unanswerable questions are not detected. No similarity threshold was defensible on this data, so none is claimed
Answer quality is not measured; only one local model is available and grading its own output would be circular
One jurisdiction, one peril, three documents. Results do not transfer to insurance documents generally
Kubernetes manifests are authored and CI-validated but have never been applied to a cluster
A fine-tuning notebook exists but is archived: its training data was the earlier evaluation set, so no tuned-model result could be honest

What this changed in my practice

A benchmark is only worth what its hardest cases are worth. Near-duplicate documents exposed a failure that topic-level retrieval metrics would have hidden entirely.

Source and documentation Discuss relevant work