Current engineering / Illustrative model
Anything derived should be rebuildable. Anything else must not be overwritten.
Most data platforms end up holding the same information several times over — as records, as vectors, as a graph. Each copy earns its place, and each one can quietly stop agreeing with the others. This is a synthetic model of how to keep them honest, and of the engineering I work on.
An illustrative model of a class of problem, not a description of any deployed system. Every quantity is synthetic and chosen for legibility. No architecture, technology, identifier, or scale from any employer system appears anywhere in it.
01The problem
Four descriptions of one thing, three of them disposable.
One capture, several derived views. They are not stages in a pipeline; they exist at the same time and each is good at something the others are bad at. Records answer exactly. A semantic index answers approximately, which is the point of it. A graph keeps structure that flat records flatten away.
The captured evidence is different in kind. It is not another interpretation, it is what the interpretations are checked against, and it is the only part that cannot be regenerated from anything else.
- Captured evidenceExactly what was received, and a fingerprint taken at the moment of capture.
- Structured recordsOrdered records with stable identities and explicit fields.
- Semantic indexPassages placed in a space where distance stands for similarity.
- Relationship graphEntities and the connections between them.
02Verification
The interesting arrow points backwards.
A diagram of a system like this usually runs one way and stops at the output. The question worth engineering for is the other direction: does what we stored still agree with what it was built from? Each derived view gets asked, and the answer is a conjunction rather than a light.
- PresentIs the captured evidence still there, byte for byte?The one thing that cannot be rebuilt from anything else.
- CompleteDoes everything that should have been derived actually exist?A partial build looks healthy from any single store.
- ConsistentDo the derived representations still agree with the authoritative record?Each one can drift on its own schedule.
- CurrentIs the upstream source still saying what we captured?Nothing local changes when an external source does.
03Rules that make a check mean something
Most of the difficulty is ownership, not measurement.
- One writer per piece of stateIf two components can write the same thing, neither can be held to it, and no check over it means anything.
- Whatever measures is not whatever reportsA service that runs the checks and also publishes the verdict is marking its own homework. The reporting layer reads results it did not produce.
- The reporting layer writes nothing it reports onIt reads every representation and owns none of them, so looking at the system cannot change it.
- Nothing runs on a timerEvery action begins with a person deciding to take it. A schedule hides the moment a decision was made.
- Derived state is disposable; captured evidence is notAnything that can be rebuilt from evidence should be rebuildable on demand, and anything that cannot be rebuilt must never be overwritten.
04When it goes wrong
The failures worth designing for are the quiet ones.
None of these describes an incident. They are the classes of failure this shape of system has, and the reason the checks above are worth running: each of them looks perfectly healthy from inside the store it affects.
The response is an operator action rather than a scheduled job. Derived state is discarded and rebuilt from evidence; the evidence itself is never touched.
- Stale derived stateA representation was built from an earlier version and never rebuilt.
- Incomplete derivationA build stopped part-way and left a representation that looks whole.
- Silent divergenceTwo representations disagree and nothing compares them.
- Upstream changeThe source changed after capture, so what is held is correct and no longer current.
- VerifyRe-runs the checks and refreshes the verdicts. Writes no content.
- RebuildDiscards the derived representations for one item and reconstructs them from captured evidence.
05What I worked on
Stated as problems, and only where I can show the commits.
- Operator-facing health and status: turning a set of invariants into a verdict someone can act on, including which term failed.
- Cross-representation reconciliation: comparing derived representations against the authoritative record rather than trusting each in isolation.
- Source currency checking: detecting when an upstream source has moved on from what was captured.
- Rebuild workflows: discarding and reconstructing derived state from captured evidence, as an operator action rather than a scheduled job.
- Extraction and entity work upstream of all of it, on a smaller share of the code.
Boundaries
What this page is not.
- Not a deployment topologyNo service, store, queue, or interface from any real system appears here.
- No production scaleEvery quantity is chosen to be legible on a screen and implies nothing about any deployment.
- No internal dataNo content, identifier, schema, or fingerprint from any employer system is used or reproduced.
- A class of problemThe engineering principles are real. The machine that illustrates them is invented for this page.
The engineering above is real and is what I spend my time on. The machine that illustrates it was invented for this page, and every quantity in it was chosen because it is legible on a screen.