Retrieval on the JVM

Two Implementations, One Ranking

A search scorer rewritten from Python to Java.

Role: Project author
Classification: Reference implementation

One real ranking from the parity fixture. The top two are four thousandths apart, which is the pair a different summation order would swap.

Problem
A rewrite is easy to get almost right.
How it was solved
I tested the Java output against the Python, query by query.
1. Python
2. Java
3. Parity test
Result
50
real queries match exactly
Built with
- Java 21
- Maven
- JUnit 5
See the evidence on GitHub

Reference implementation / Retrieval

Rewriting BM25 in another language is not the hard part. Getting it to return the same order, when two of the scores differ in the fourth decimal, is.

Rankings compared exactly50 questions

Chunked on the JVM, compared against an exported ranking

Chunking is done here rather than read from the Python fixture, or the parity test would pass without exercising the chunker. Order is compared exactly; only the scores get a tolerance.

This diagram in words

Checksummed corpus source — Verified first, so both implementations provably read the same bytes.
Chunk and tokenise process — Done on the JVM, not loaded from the fixture.
Java 21
BM25 index process — Okapi scoring with the reference implementation's k1 and b.
Rank fusion process — By position, so two score scales never have to be reconciled.
Exported reference source — The Python ranking, committed as JSON.
Jackson
Parity test gate — Order exactly. Scores only to a tolerance.
JUnit 5

01 / 7One question, five hitsThe first question in the fixture, and the chunks BM25 ranks for it.

1. dwelling b1e73cc5: 18.337068
2. general property 823f35c2: 18.332857
3. rcbap 032a6c69: 16.510989
4. rcbap d026af40: 15.009405
5. dwelling e5de6486: 14.320981

The first question in the committed parity fixture and its five BM25 hits, drawn as bars in rank order with their scores to six decimal places: 18.337068 for a dwelling-form chunk, 18.332857 for a general-property one, then 16.510989, 15.009405 and 14.320981. The top two lift apart as the pair most at risk of swapping, and a magnified window opens beneath the bars - the bars themselves are never rescaled - showing the gap between them as a visible distance and labelling it 0.004211, about four thousand times smaller than the largest score on the chart. The rules the parity test applies follow: order is compared exactly, scores are allowed a relative tolerance of 1e-12. Then the limiting case, a question elsewhere in the fixture where two chunks score identically to every digit a double carries. The sequence closes on the scope of the comparison - fifty questions, 314 chunks, k1 1.5 and b 0.75 - and on the corpus it was built from, the NFIP standard flood insurance policy forms at 44 CFR Part 61.

Read the full background

Problem

Why this work exists

Rewriting a retrieval scorer in another language is easy to do approximately and hard to do identically. Floating-point addition is not associative, so summing the same contributions in a different order can land a few units in the last place away, and on this corpus the top two results for the first question are about four thousandths apart. That is close enough for a reordering to look like a bug in the ranking rather than a difference in arithmetic.

Contribution

What I can claim

Implemented the tokenizer, chunker, BM25 index and rank fusion with no dependencies in the main source, then wrote the parity test that loads the corpus, verifies it against a pinned SHA-256, chunks it independently and compares its own output against the Python reference rather than trusting the exported fixture.

Method

Proving two implementations agree before trusting either

A search implementation in one language is usually checked against intuition.

The top two are 0.004211 apart. Summing the same contributions in a different order would swap them, so the Java side keeps the Python loop order and both return this ranking for all 50 questions.

Verify
The corpus is checked against its checksum first.
Chunk
Text is split on the JVM rather than loaded from a fixture, so the first thing compared is the step most likely to differ and least likely to be looked at.
Score
The index is built and the ranking function applied.
Fuse
Ranked lists are combined by position rather than by score, which is what makes the comparison robust to the small numerical differences the previous step tolerated.
Expose
The closest pair of scores in the whole corpus is reported, because that pair is the one a smaller tolerance would break - and naming it is more useful than asserting the test passes.
What this does not show
Parity is against one reference implementation on one corpus. It establishes that the two agree, not that either is correct, and a shared misreading of the algorithm would pass every check here.

How it was built

System

Workflow and decisions

The test is the artefact. It verifies the corpus, chunks it independently, builds an index, fuses two ranked lists, and compares order against a committed reference produced by a different language's implementation of the same specification.

1Verify the corpus against its checksum
2Chunk it on the JVM, not from the fixture
3Build the index and score
4Fuse the ranked lists by position
5Compare order exactly, scores to a tolerance

Java 21
Maven
JUnit 5
Jackson
GitHub Actions

Evidence

What is actually versioned

Rankings compared exactly50 questions

BM25 order and identifiers at depth 20, fused order at depth 10, both compared exactly rather than to a tolerance. Scores are allowed a relative 1e-12.

Chunking compared firstall 314 chunks

Count, identifiers, offsets and text, across three corpus files. Java chunks the corpus itself rather than reading the exported chunks, so a chunking difference fails as a chunking mismatch instead of surfacing later as a confusing BM25 discrepancy.

The pair most at risk0.004211 apart

The top two results for nfip-001 differ by that much. A different summation order would swap them, which is why the loop order matches the reference, an inverted index is deliberately not used, and the logarithm is StrictMath rather than Math.

How the work is checked

Quality controls

How the work is checked

No dependencies in the main source: BM25 and rank fusion need arithmetic, a hash and a regex, and a retrieval library would hide the thing the repository exists to show
The corpus is verified against a SHA-256 in its own manifest before anything is indexed
Chunking is compared before scoring, so failures surface at the layer that caused them
StrictMath rather than Math, because the latter is permitted a unit in the last place of error and may be swapped for a JIT intrinsic
Repeated query terms are not collapsed, because the reference does not collapse them either

Limitations

Where the evidence stops

Parity is against one reference implementation on one corpus. It shows the two agree, not that either is correct
The dense vectors are an exported fixture rather than recomputed on the JVM, and the embedding model is pinned by name only, so re-exporting on a different stack can change those lists
Fifty questions over three near-duplicate policy forms is a parity harness, not a retrieval benchmark, and no relevance judgement is involved
A library-free implementation chosen for comparability rather than for speed: the slower document-major loop is the correct one here

What this changed in my practice

Agreement between two implementations is only meaningful if the comparison is exact, and making it exact meant giving up the obvious optimisation. An inverted index would visit documents in posting-list order, which reorders the additions, which moves the last few bits, which swaps a pair four thousandths apart.

Next evidence milestone: Recompute the dense side on the JVM. The sparse half is held to the reference exactly; the dense half is a fixture, so the fused ranking is only as reproducible as an embedding model pinned by name.

Source and documentation Discuss relevant work