Water quality classification
The Ranking the Run Does Not Support
A fair comparison of four classifiers.
- Role
- Author
- Classification
- Synthetic demonstration
Problem
One score per model suggests a ranking the data cannot support.
How it was solved
I compared the spread across five folds, not one score.
- Five folds
- Four models
- Compare spread
Result
1 of 4
models truly separated
- Logistic regression0.693
- Random forest0.787
- XGBoost0.796
- SVM0.819
Built with
- scikit-learn
- XGBoost
- pandas
Synthetic demonstration / Classification
Five models, a table ordering them by score, and a clear winner in bold. The same run also recorded how much each model moved between folds - so how many of those places are real?
Models the run separates1 of 4
Five estimators, one tuned, and a ranking that does not hold
This diagram in words
- Seeded generator source — 5,000 deterministic samples. No measurement of water anywhere.
- Impute and scale process — Median imputation; standard scaling only where the estimator needs it.pandas · scikit-learn
- Five-fold CV process — Stratified, on the training split, scored by F1.scikit-learn
- Grid search process — Twenty-seven configurations, and only for this one estimator.XGBoost
- Ordered table gate — Three contenders' intervals overlap, so the order is not supported.
- Committed metrics artifact — Written to a tracked file, so the table cannot drift from the run.
Read the full background
Problem
Why this work exists
A model comparison that reports one score per model invites a reading it cannot support. Fold-to-fold variation is usually larger than the gaps being ranked, so the order is partly an artefact of which split was drawn - and the table that gets copied into a README is exactly the artefact that hides this.
Contribution
What I can claim
Built the pipeline and the comparison, then read the tracked artefact against its own headline table. The cross-validated F1 and its standard deviation are recorded per model in `results/model_results.json`; drawing them as intervals is what turns a ranking back into a measurement with a width.
Method
What a leaderboard hides from itself
Four models, five rows, a winner in bold.
- 0.819
- 0.796
- 0.787
- 0.693
- —
The shaded column is the range consistent with all 3 intervals at once, 0.804 to 0.809. The axis begins at 0.65 rather than zero so the intervals are legible, and the data is generated rather than measured: this says nothing about whether any real water is safe.
Generate
The dataset is produced by a seeded generator with deliberately class-correlated features.
Compare
Each model runs inside a pipeline, so imputation and scaling are fitted per fold rather than once over everything.
Rank
Scored on a single held-out split, the models fall into a clean order with a clear best.
Widen
Each model also has a standard deviation across folds.
Concede
So the finding is narrower than the table: one model is clearly worse than the others, and the order of the rest is not established.
What this does not show
The labels and features are generated, not laboratory measurements, and the repository states it does not predict whether real water is safe to drink and must not be used for water-quality or regulatory decisions. The intervals here are fold-to-fold standard deviations from a single seeded run, which is a floor on the uncertainty rather than a confidence interval.
How it was built
System
Workflow and decisions
Two modules: a seeded generator and one pipeline that runs preprocessing, the four classifiers, cross-validation, a 27-point grid search for XGBoost and the evaluation plots. Metrics land in a tracked JSON file, which is what lets this page quote them without re-running anything.
- 1Generate 5,000 samples from the seeded generator
- 2median imputation for the missing values
- 3standard scaling, for the models that need it
- 4five-fold stratified cross-validation on the training split
- 5a 27-point grid search, for XGBoost only
- 6Track the metrics as a committed artefact rather than a screenshot
- Python
- scikit-learn
- XGBoost
- pandas
- matplotlib
Evidence
What is actually versioned
Taking the cross-validated F1 of each model plus or minus one standard deviation across 5 folds, the intervals for the top 3 all overlap: a single value between 0.804 and 0.809 is consistent with every one of them. Only Logistic Regression sits clear, and it sits clear by 0.094. The ordering of the other three is not something this run established.
Random Forest moves 0.0221 between folds, and the closest pair of contenders is 0.0084 apart. When the noise is more than twice the difference, the difference is not a finding - and no interval at all was recorded for XGBoost (tuned), which the headline table nonetheless lists among the leaders.
All 5 rows measure how well a model recovers the deliberately class-correlated distributions of a seeded generator, at seed 42, with 7% of values missing in pH, sulfate, trihalomethanes. The repository states it does not predict whether real water is safe to drink and must not be used for water-quality or regulatory decisions.
How the work is checked
Quality controls
How the work is checked
- The metrics are a committed artefact from a seeded run, so the figures on this page are computed from it rather than transcribed, and a changed run changes the page
- Imputation and scaling sit inside the scikit-learn pipelines, so they are fit per fold rather than once over the whole training set
- Cross-validation is stratified, which matters when the generated classes are split sixty-forty
- The README says in its first lines that the labels are generated and that the repository must not be used for water-quality decisions, rather than leaving the caveat to the bottom of the page
Limitations
Where the evidence stops
- The data is generated. A model that recovers a generator's own correlations has demonstrated that the pipeline runs, not that anything can be predicted about water
- One seeded split and five folds. The intervals here are the fold-to-fold standard deviation, which is the smallest honest account of the uncertainty rather than a confidence interval
- No cross-validation statistics were recorded for XGBoost (tuned), so it cannot be compared with the rest on the same footing
- The nine features are described as illustrative and are explicitly not a standards-compliant representation of any authority's criteria
What this changed in my practice
What this changed in my practice
Writing the comparison table was quick; reading it honestly took longer. The artefact already contained the standard deviations that undercut its own ordering, which is the uncomfortable and useful part - the evidence against the headline was sitting in the same file as the headline, and nothing but drawing it made that visible.