ML systems engineering
A Testable End-to-End MLOps Pipeline
A pipeline that stops a weak model from shipping.
- Role
- Project author and engineer
- Classification
- Reference implementation
Problem
A perfect score came from duplicated rows.
How it was solved
I added real data, a held-out test and a gate that can refuse a release.
- Validate
- Train
- Gate
- Serve
Result
0.8067
held-out accuracy on licensed data
- Baseline0.5000
- Model0.8067
Built with
- scikit-learn
- FastAPI
- Docker
- pytest
- GitHub Actions
Reference implementation / Delivery
Promotion is not the next box along a diagram. It is a conjunction of four checks, and one short check stops the candidate where it stands.
Held-out accuracy0.8067
One fitting surface, and a gate that stops the run
This diagram in words
- Licensed data source — Hashed and checked for duplicates before anything reads it.
- Partition process — Train, validation and test, fixed before a feature exists.
- Fit transformers process — The only place anything is fitted. Train partition only.scikit-learn
- Transform process — Cannot fit. Validation, test and serving share this path.
- Train candidate process — Produces a bundle with checksummed artifacts.scikit-learn
- Promotion gate gate — A conjunction of checks on held-out metrics and latency.
- Registry store — Registered as staging, then transitioned to production.
- Prediction service sink — Loads only a bundle whose gate flag and checksums verify.FastAPI
- Gate failure sink — The run raises and stops. Nothing is registered, and nothing retries.
Read the full background
Problem
Why this work exists
The first version of this pipeline had every stage in place and proved nothing. Its only data path generated review text from ten templates, so the classifier scored a perfect 1.000 on rows that were 99% duplicates. The validator detected those duplicates, logged a warning, and let the run continue. A gate that cannot fail is not a gate, and a metric produced that way describes the fixture rather than the model.
Contribution
What I can claim
Replaced the synthetic evidence path with a checksum-verified licensed dataset, split it three ways so the test partition is read exactly once, restricted feature fitting to a single function so leakage is testable rather than asserted, added a majority-class baseline and a gate that requires a margin over it, and made the container prove a real prediction in CI instead of only building.
- 01ValidateCheck, fingerprint, and version input data
- 02TransformFit features once for training and serving
- 03EvaluateTrack runs and apply the promotion gate
- 04RegisterPackage an immutable model bundle
- 05ServeLoad the approved bundle behind FastAPI
$ python -m src.pipeline --config configs/train_config.yaml
[1/5] Acquiring and validating data
[2/5] Fitting features on train only, then training
[3/5] Evaluating on the held-out test split
[4/5] Registering the passing candidate as staging
[5/5] Promoting v1 to production
$ python scripts/check_reference_run.py
Reference run matches the documented result: accuracy 0.8067, baseline 0.5000, margin 0.3067,
sentiment-classifier v1 in production.The second command exists so the numbers quoted on this page cannot drift away from the repository without failing its build. See the evaluation record.
Confusion matrix
| pred negative | pred positive | |
|---|---|---|
| actual negative | 241 | 59 |
| actual positive | 57 | 243 |
484 of 600 correct. The errors are close to symmetric, so the model is not simply favouring one class.
Accuracy against its baseline
An accuracy figure means nothing without this comparison. The promotion gate requires a margin over the baseline, not just a raw floor, so a model that only predicts the larger class cannot pass.
Method
Promotion is conditional, and the pipeline has to be able to refuse
A pipeline that always finishes is not a pipeline, it is a conveyor.
- Verify and validate licensed data
- Split train / validation / test
- Fit features on train only
- Evaluate held-out and gate
- Register, promote, and serve
Ingest
Data enters under a licence the pipeline can verify.
Separate
Train, validation and test are split before anything is fitted, so the held-out set is genuinely held out rather than nominally so.
Fit
Features are fitted on the training split alone.
Gate
The held-out result is compared against a threshold.
Promote
Only an artifact that passed is registered and served, and what it passed is recorded alongside it.
What this does not show
The gate proves the artifact met a threshold on one held-out split at one moment. It does not establish that the split is representative, that the threshold is the right one, or that the model will hold up on data collected later.
How it was built
System
Workflow and decisions
Each stage is a contract. Data is verified against a pinned checksum and can be refused, preprocessing is fitted on the training partition alone, evaluation reconstructs the split recorded in the candidate's lineage rather than accepting one supplied later, and the service loads only a checksum-valid bundle the registry marks as production.
- 1Verify and validate licensed data
- 2Split train / validation / test
- 3Fit features on train only
- 4Evaluate held-out and gate
- 5Register, promote, and serve
- scikit-learn
- FastAPI
- Docker
- pytest
- GitHub Actions
- ruff
Evidence
What is actually versioned
Weighted F1 0.8067 on 600 test rows, against a majority-class baseline of 0.5000, a margin of 0.3067.
PR-AUC 0.8895 on the same held-out rows.
UCI Sentiment Labelled Sentences, 1,500 positive / 1,500 negative, CC BY 4.0. Downloaded on demand against a pinned SHA-256 and never redistributed.
Train, validation, test. Gate thresholds were derived from the validation split and the baseline; the test split was not used to select a threshold, model, or hyperparameter.
The same model and transformer files, and metrics agreeing to twelve decimal places, across Windows, a clean virtual environment, and Ubuntu CI.
Unit, integration, and container layers. The container test builds the image, mounts a promoted bundle, and makes a real HTTP prediction.
Inspection points
Go directly to the evidence
How the work is checked
Quality controls
How the work is checked
- Input validation can refuse: a degenerate dataset stops the run instead of producing a flattering score on it
- One function fits preprocessing state; a test asserts that a token seen only outside training never enters the vocabulary
- The promotion gate requires a margin over a majority-class baseline, so a raw accuracy floor cannot pass an imbalanced non-model
- A published bundle carries SHA-256 checksums for 5 artifacts plus dataset provenance and licence, all validated before loading
- The service separates liveness from readiness and starts unready rather than crashing when no model is available
- CI runs offline tests, the licensed-data reference run, and container integration as separate jobs
Limitations
Where the evidence stops
- Only 600 held-out test rows, so the confidence interval on accuracy is roughly three points; a one-point difference between models on this split is noise
- Metrics are pooled across the dataset's three sources; per-source performance is not reported and would very likely differ
- No probability calibration is reported, because nothing downstream consumes the probabilities as probabilities
- A reference implementation that has never carried production traffic
- Monitoring is an in-process counter endpoint over a bounded window, not a monitoring system
- No delayed-label path exists, so nothing measures accuracy after deployment, only behaviour
- MLflow tracking is optional and disabled by default; the bundle, not a tracking server, is the source of truth for promotion
What this changed in my practice
What this changed in my practice
A pipeline can have every correct stage and still prove nothing. The evidence a run produces is only as good as the data underneath it, and the fastest way to find that out is to give the quality gate something it can actually refuse.