Current engineering / Illustrative model
Anything derived should be rebuildable. Anything else must not be overwritten.
Most data platforms end up holding the same information several times over — as records, as vectors, as a graph. Each copy earns its place, and each one can quietly stop agreeing with the others. This is a synthetic model of how to keep them honest, and of the engineering I work on.
An illustrative model of a class of problem, not a description of any deployed system. Every quantity is synthetic and chosen for legibility. No architecture, technology, identifier, or scale from any employer system appears anywhere in it.
Problem
Copies made from one source can drift apart silently.
How it was solved
Each copy is checked backwards against the original capture.
- Capture
- Derive
- Verify back
Result
Backwards
every copy is traced to its source
Built with
The named systems are on the architecture site.
See the evidence
Current engineering / Synthetic model
One captured source becomes an index, a graph and a cache. Days later, is any of them still telling the truth about it?
Public-safe modelSynthetic throughout
How it works
One capture, checked from the far end
Several derived views of one source exist at the same time, and each is good at something the others are bad at. The engineering question is whether they still agree with what they were built from.
Three derived forms, each checked against the capture
Problem
Derived views are disposable. The captured evidence is not: it is what they are checked against, and the only part that cannot be regenerated.
Derive
One capture is projected into forms with different jobs. Records answer exactly, a semantic index answers approximately, and a graph keeps structure that flat records lose.
Compare
Checking the derived forms against each other finds disagreement without identifying who is wrong.
Verify
So the check runs backwards, asking each derived view whether it still agrees with the capture. The answer is a conjunction rather than a light.
What this does not show
This is a synthetic model. It is illustrative throughout and describes no deployed system.
The same work, named
This diagram in words
- Captured source source — Illustrative throughout. The retained bytes are the evidence.
- Derived three ways process — One capture projected into three representations with different jobs.
- Index store — For finding a unit without reading everything.
- Graph store — For following references between units.
- Cache store — For answering the same question again cheaply.
- Checked backwards gate — From each derived form to the evidence, not forward from it.
- Rebuilt from evidence sink — A form that has drifted is made again rather than corrected in place.
The problem
Four descriptions of one thing, three of them disposable.
One capture, several derived views. They are not stages in a pipeline; they exist at the same time and each is good at something the others are bad at. Records answer exactly. A semantic index answers approximately, which is the point of it. A graph keeps structure that flat records flatten away.
The captured evidence is different in kind. It is not another interpretation, it is what the interpretations are checked against, and it is the only part that cannot be regenerated from anything else.
- Captured evidenceExactly what was received, and a fingerprint taken at the moment of capture.
- Structured recordsOrdered records with stable identities and explicit fields.
- Semantic indexPassages placed in a space where distance stands for similarity.
- Relationship graphEntities and the connections between them.
Verification
The interesting arrow points backwards.
A diagram of a system like this usually runs one way and stops at the output. The question worth engineering for is the other direction: does what we stored still agree with what it was built from? Each derived view gets asked, and the answer is a conjunction rather than a light.
Ingest
Represent
Verify
Observe
- PresentIs the captured evidence still there, byte for byte?The one thing that cannot be rebuilt from anything else.
- CompleteDoes everything that should have been derived actually exist?A partial build looks healthy from any single store.
- ConsistentDo the derived representations still agree with the authoritative record?Each one can drift on its own schedule.
- CurrentIs the upstream source still saying what we captured?Nothing local changes when an external source does.
Rules that make a check mean something
Most of the difficulty is ownership, not measurement.
- One writer per piece of stateIf two components can write the same thing, neither can be held to it, and no check over it means anything.
- Whatever measures is not whatever reportsA service that runs the checks and also publishes the verdict is marking its own homework. The reporting layer reads results it did not produce.
- The reporting layer writes nothing it reports onIt reads every representation and owns none of them, so looking at the system cannot change it.
- Nothing runs on a timerEvery action begins with a person deciding to take it. A schedule hides the moment a decision was made.
- Derived state is disposable; captured evidence is notAnything that can be rebuilt from evidence should be rebuildable on demand, and anything that cannot be rebuilt must never be overwritten.
When it goes wrong
The failures worth designing for are the quiet ones.
None of these describes an incident. They are the classes of failure this shape of system has, and the reason the checks above are worth running: each of them looks perfectly healthy from inside the store it affects.
The response is an operator action rather than a scheduled job. Derived state is discarded and rebuilt from evidence; the evidence itself is never touched.
- Stale derived stateA representation was built from an earlier version and never rebuilt.
- Incomplete derivationA build stopped part-way and left a representation that looks whole.
- Silent divergenceTwo representations disagree and nothing compares them.
- Upstream changeThe source changed after capture, so what is held is correct and no longer current.
- VerifyRe-runs the checks and refreshes the verdicts. Writes no content.
- RebuildDiscards the derived representations for one item and reconstructs them from captured evidence.
What I worked on
Stated as problems, and only where I can show the commits.
- Operator-facing health and status: turning a set of invariants into a verdict someone can act on, including which term failed.
- Cross-representation reconciliation: comparing derived representations against the authoritative record rather than trusting each in isolation.
- Source currency checking: detecting when an upstream source has moved on from what was captured.
- Rebuild workflows: discarding and reconstructing derived state from captured evidence, as an operator action rather than a scheduled job.
- Extraction and entity work upstream of all of it, on a smaller share of the code.
Boundaries
What this page is not.
- Not a deployment topologyNo service, store, queue, or interface from any real system appears here.
- No production scaleEvery quantity is chosen to be legible on a screen and implies nothing about any deployment.
- No internal dataNo content, identifier, schema, or fingerprint from any employer system is used or reproduced.
- A class of problemThe engineering principles are real. The machine that illustrates them is invented for this page.
The engineering above is real and is what I spend my time on. The machine that illustrates it was invented for this page, and every quantity in it was chosen because it is legible on a screen.