ML systems engineering

A Retraining Loop That Watches the Target

A forecasting service that spots when it goes stale.

Role: Project author and engineer
Classification: Reference implementation

Two drift scores on one scale, each against the threshold it is tested against. The features stop short of their line; the target runs well past its own.

Problem
Watching only the inputs would miss the change.
How it was solved
It watches the target too, and a gate decides on retraining.
1. Monitor
2. Retrain
3. Gate
Result
0.679 vs 0.09
target drift against input drift
- Target drift0.679
- Input drift0.09
Built with
- scikit-learn
- MLflow
- Evidently
- Airflow
See the evidence on GitHub

Reference implementation / Monitoring

The monitor you would build first watches the inputs. Here the inputs barely move, and the thing being predicted moves enormously.

Where the drift actually was0.679 against 0.09

A retraining loop that watches what it predicts

The monitors read production traffic, so they sit after the service rather than beside the data. The two signals are separate because they disagree - the inputs barely move and the target moves a long way past its threshold.

This diagram in words

Pinned dataset source — Fixed by SHA-256, so a run can name the data it used.
Challenger gate gate — Must beat the champion re-scored on the same window. Offered itself, it is refused.
scikit-learn · MLflow
Champion alias store — Moving this pointer is the deployment. Nothing is copied anywhere.
MLflow
Prediction service sink — Loads whatever the alias currently names.
FastAPI
Feature drift process — Serving window against the same seasons a year earlier.
Evidently
Target drift process — What is being predicted, which moves far more than the inputs.
Retrain decision gate — Either signal crossing its own threshold is enough.
Airflow

01 / 7The first model clears the baselineMAE 51.49 against a baseline of 98.94. Nothing is registered yet, so it is promoted.

Train on 2011: MAE 51.49 — promoted
Retrain including 2012: MAE 95.48 — promoted
Offer the same model again: MAE 95.48 — refused
Why the third is refused: challenger MAE 95.48 is not 2% better than champion 95.48 (needed <= 93.57)
Champion re-scored on the new window: 116.57

One axis of 8,734 observations in the order they arrived, cut eight different ways. The 2011 control window is marked across the first 1,729 rows. Eleven feature-column lanes draw across it and one, humidity, lights: it is the only column that crossed its own 0.1 threshold, which is why the feature share is 0.09. Two drift scores then lift onto a common scale against the thresholds the policy compares them with - the features stop short of 0.1, the target runs to 0.679 against a threshold of 0.2. The ADWIN detector is silent for every row of the control window and raises its first alarm at row 1,855. Three training runs are drawn over the windows they were fitted to: 1,729 rows scoring MAE 51.49 and promoted, 3,476 rows scoring 95.48 and promoted over a champion re-scored at 116.57, and the same model offered again and refused because 95.48 is not two per cent better than 95.48, which would have needed 93.57. Four candidate window widths follow, with the two narrowest marked as the defect: a 500-row window is twenty-one days of January being compared against a reference spanning twelve months. After the seasonal alignment fix the day twenty-one false alarm declines and the real signal arrives on day 42 rather than day 104.

Read the full background

Problem

Why this work exists

The obvious way to monitor a deployed model is to watch its input features for drift. On this dataset that design is almost silent. Between 2011 and 2012 one feature column of eleven crosses its threshold, a feature drift share of 0.09, while the target's distribution moves 0.679. A monitor watching inputs alone would have reported nothing worth acting on.

Contribution

What I can claim

Built the loop and its policy: a drift check that measures the target first because that is the signal that fires here, two promotion gates a challenger has to clear, and a split between measurement and policy so the policy can be unit tested without a library, a network or any data.

Method

Knowing when a model has gone stale

A deployed model degrades without announcing it.

A monitor watching the inputs alone would have declined to act.

Pin
The dataset is fixed by checksum before anything is fitted, so every later comparison is against a known input rather than against whatever the source happened to be serving.
Compare
A serving window is held against the same seasons a year earlier, not against the window immediately before it.
Detect
Two signals can trigger a retrain, because the distribution of the inputs moving and the relationship itself moving are different failures and either one is enough.
Gate
A challenger is fitted and then has to earn promotion.
Promote
Only then does the alias the service loads move.
What this does not show
The drift score here is label shift - a distance between two marginal distributions of the target. It is a proxy for the relationship moving rather than a measurement of it, and real concept drift is not monitored directly. A streaming detector was also tried and was not faster than the windowed comparison, which is recorded rather than omitted.

How it was built

System

Workflow and decisions

Three loops that only meet through artefacts. Training fits a challenger and ships it only through two gates. Monitoring compares a serving window against the same seasons a year earlier and applies the policy. Serving loads whichever version the champion alias points at, and appends its traffic to the log the drift job reads.

1Pin the dataset by SHA-256
2Fit a challenger and gate it
3Compare a serving window against the same seasons a year earlier
4Decide retrain, on either signal
5Move the champion alias the service loads

scikit-learn
MLflow
Evidently
Airflow
FastAPI
Prometheus
Terraform

Evidence

What is actually versioned

Where the drift actually was0.679 against 0.09

Target drift against feature drift share, comparing 2012 with the same seasons in 2011. Only humidity crossed the 0.1 per-column threshold, one of 11 feature columns.

What that drift costMAE 116.57 to 95.48

The 2011 champion re-scored on the 2012 window against a model fitted to it. An old mapping fitting worse than a new one is the stronger evidence that the relationship moved, rather than the drift score itself.

A gate that refusesrefused

The exported run record carries the reason verbatim: "challenger MAE 95.48 is not 2% better than champion 95.48 (needed <= 93.57)". The same model, offered twice, is turned down the second time for not being better than itself.

A streaming detector was not fasterrow 1855

ADWIN at delta 0.00001, the most sensitive setting that raises 0 alarms on the control window, first speaks at row 1855 of 8734. The window monitor already in the service crosses its threshold in the same span, with one fewer dependency.

How the work is checked

Quality controls

How the work is checked

Measurement and policy are separate, so the policy has unit tests that need no library, no network and no data
The window guard runs before the drift check, because a handful of rows fails a distribution test for reasons that have nothing to do with the world moving
A refused promotion writes its reason to the run summary, so it is auditable rather than mysterious
The dataset is pinned by SHA-256 and downloaded on demand rather than committed
The architecture diagram is generated from the code that describes it, not drawn by hand
Both Airflow DAGs have been run, and the transcript is in the repository

Limitations

Where the evidence stops

The target drift score is label shift, a distance between two marginal distributions of the target. It is a proxy for the relationship moving rather than a measurement of it; real concept drift is not monitored directly
Nothing here produces a confidence interval on a prediction. The word calibrated in this repository refers to one thing only, the streaming detector's delta parameter
A reference implementation on a public dataset that has never carried production traffic
The drift comparison is between two years of one dataset, so the seasonal alignment that fixes the false alarm is tuned to data with a yearly cycle
No delayed-label path exists, so the loop measures distributions rather than accuracy after deployment

What this changed in my practice

The monitor you would build first is the one that would have missed this. Watching inputs is the obvious design and it is nearly silent here, and the gate that was supposed to protect production was comparing two numbers measured on different windows, which made it block the retraining that drift had just correctly called for.

Next evidence milestone: A delayed-label path. Everything here measures distributions, which is what is available at serving time, but the honest signal is accuracy against outcomes once they arrive. Until that exists the loop can tell that the world moved and cannot tell whether the retrained model is actually better in production. The seasonal fix brought the real signal forward from about day 104 to day 42, and the next thing worth measuring is what happens after the retrain.

Source and documentation Discuss relevant work