ML systems engineering
A Testable End-to-End MLOps Pipeline
A runnable reference for the lifecycle around a text classifier, rebuilt on a licensed dataset so the pipeline's own quality gate has something real to refuse.
- Role
- Project author and engineer
- Classification
- Reference implementation
01Problem
Why this work exists
The first version of this pipeline had every stage in place and proved nothing. Its only data path generated review text from ten templates, so the classifier scored a perfect 1.000 on rows that were 99% duplicates. The validator detected those duplicates, logged a warning, and let the run continue. A gate that cannot fail is not a gate, and a metric produced that way describes the fixture rather than the model.
02Contribution
What I can claim
Replaced the synthetic evidence path with a checksum-verified licensed dataset, split it three ways so the test partition is read exactly once, restricted feature fitting to a single function so leakage is testable rather than asserted, added a majority-class baseline and a gate that requires a margin over it, and made the container prove a real prediction in CI instead of only building.
- 01ValidateCheck, fingerprint, and version input data
- 02TransformFit features once for training and serving
- 03EvaluateTrack runs and apply the promotion gate
- 04RegisterPackage an immutable model bundle
- 05ServeLoad the approved bundle behind FastAPI
$ python -m src.pipeline --config configs/train_config.yaml
[1/5] Acquiring and validating data
[2/5] Fitting features on train only, then training
[3/5] Evaluating on the held-out test split
[4/5] Registering the passing candidate as staging
[5/5] Promoting v1 to production
$ python scripts/check_reference_run.py
Reference run matches the documented result: accuracy 0.8067, baseline 0.5000, margin 0.3067,
sentiment-classifier v1 in production.The second command exists so the numbers quoted on this page cannot drift away from the repository without failing its build. See the evaluation record.
Confusion matrix
| pred negative | pred positive | |
|---|---|---|
| actual negative | 241 | 59 |
| actual positive | 57 | 243 |
484 of 600 correct. The errors are close to symmetric, so the model is not simply favouring one class.
Accuracy against its baseline
An accuracy figure means nothing without this comparison. The promotion gate requires a margin over the baseline, not just a raw floor, so a model that only predicts the larger class cannot pass.
Method
Promotion is conditional, and the pipeline has to be able to refuse
A pipeline that always finishes is not a pipeline, it is a conveyor. What makes this one a reference is that each stage can decline to pass the artifact on, and the refusal is the interesting behaviour.
- Verify and validate licensed data
- Split train / validation / test
- Fit features on train only
- Evaluate held-out and gate
- Register, promote, and serve
Ingest
Data enters under a licence the pipeline can verify. A gate that cannot check its input is decoration.
Separate
Train, validation and test are split before anything is fitted, so the held-out set is genuinely held out rather than nominally so.
Fit
Features are fitted on the training split alone. This is where leakage usually enters, quietly, and never announces itself later.
Gate
The held-out result is compared against a threshold. If it falls short the artifact stops here - it is not registered, not promoted, and not served.
Promote
Only an artifact that passed is registered and served, and what it passed is recorded alongside it.
What this does not show
The gate proves the artifact met a threshold on one held-out split at one moment. It does not establish that the split is representative, that the threshold is the right one, or that the model will hold up on data collected later.
03System
Workflow and decisions
Each stage is a contract. Data is verified against a pinned checksum and can be refused, preprocessing is fitted on the training partition alone, evaluation reconstructs the split recorded in the candidate's lineage rather than accepting one supplied later, and the service loads only a checksum-valid bundle the registry marks as production.
- 01Verify and validate licensed data
- 02Split train / validation / test
- 03Fit features on train only
- 04Evaluate held-out and gate
- 05Register, promote, and serve
- scikit-learn
- FastAPI
- Docker
- pytest
- GitHub Actions
- ruff
04Evidence
What is actually versioned
Weighted F1 0.8067 on 600 test rows, against a majority-class baseline of 0.5000 — a margin of 0.3067.
PR-AUC 0.8895 on the same held-out rows.
UCI Sentiment Labelled Sentences, 1,500 positive / 1,500 negative, CC BY 4.0. Downloaded on demand against a pinned SHA-256 and never redistributed.
Train, validation, test. Gate thresholds were derived from the validation split and the baseline; the test split was not used to select a threshold, model, or hyperparameter.
The same model and transformer files, and metrics agreeing to twelve decimal places, across Windows, a clean virtual environment, and Ubuntu CI.
Unit, integration, and container layers. The container test builds the image, mounts a promoted bundle, and makes a real HTTP prediction.
05Inspection points
Go directly to the evidence
06Quality controls
How the work is checked
- Input validation can refuse: a degenerate dataset stops the run instead of producing a flattering score on it
- One function fits preprocessing state; a test asserts that a token seen only outside training never enters the vocabulary
- The promotion gate requires a margin over a majority-class baseline, so a raw accuracy floor cannot pass an imbalanced non-model
- A published bundle carries SHA-256 checksums for 5 artifacts plus dataset provenance and licence, all validated before loading
- The service separates liveness from readiness and starts unready rather than crashing when no model is available
- CI runs offline tests, the licensed-data reference run, and container integration as separate jobs
07Limitations
Where the evidence stops
- Only 600 held-out test rows, so the confidence interval on accuracy is roughly three points; a one-point difference between models on this split is noise
- Metrics are pooled across the dataset's three sources; per-source performance is not reported and would very likely differ
- No probability calibration is reported, because nothing downstream consumes the probabilities as probabilities
- A reference implementation that has never carried production traffic
- Monitoring is an in-process counter endpoint over a bounded window, not a monitoring system
- No delayed-label path exists, so nothing measures accuracy after deployment — only behaviour
- MLflow tracking is optional and disabled by default; the bundle, not a tracking server, is the source of truth for promotion
What this changed in my practice
A pipeline can have every correct stage and still prove nothing. The evidence a run produces is only as good as the data underneath it, and the fastest way to find that out is to give the quality gate something it can actually refuse.