Statistical learning coursework
Sixty Test Points and What They Can Resolve
Machine learning coursework, rebuilt and checked.
- Role
- Author
- Classification
- Reproducible experiment
- Institution
- Technical University of Munich
Problem
Reports hide the runs that went badly.
How it was solved
I kept every run, including the default that lost to guessing.
- Notebook
- Search
- Held-out test
Result
1 point
between the two best searches
- Default network23 of 60
- Majority guess32 of 60
- Random search48 of 60
Built with
- Python
- PyTorch
- NumPy
- scikit-learn
TUM coursework / Statistical learning
Backpropagation derived by hand and checked two ways, then tuned by two different searches. With sixty held-out points to judge it on, which of the differences between those runs is a difference at all?
Random search over Bayesian optimisation1 test point
One derivation, two independent checks, sixty test points
This diagram in words
- Gradients by hand source — Backpropagation derived on paper, then implemented from the derivation.NumPy
- Central differences gate — Against the function it claims to differentiate.
- Autograd gate — Against an entirely separate implementation.PyTorch
- Against the majority class gate — A wall the untouched configuration does not clear.
- Sweep, then two searches process — A learning-rate sweep, then two different tuners over the same space.scikit-learn
- Written from the run artifact — Every figure, table and quoted number emitted by the notebook itself.Jupyter · LaTeX
Read the full background
Problem
Why this work exists
A coursework report is the easiest document in the world to make look right. Figures get pasted, numbers get retyped, and the runs that embarrassed you simply do not appear - not dishonestly, just by omission, because nothing in the format requires them.
Contribution
What I can claim
Wrote the assignments so that the report could not drift from the notebook: every figure and table is written by the code, and the final assignment carries a ledger naming each quoted number and the cell that produced it. Backpropagation is derived by hand and checked against two independent references rather than trusted.
Method
What a report can be stopped from leaving out
Coursework reports are easy to make look right: figures get pasted, numbers get retyped, and the runs that went badly simply never appear.
- 23/60the default settingslearning rate 0.001, twenty thousand epochs
- 32/60guessing the majority classno model at all
- 47/60after a learning-rate sweeplearning rate 10, four orders of magnitude up
- 47/60Bayesian optimisationretrained on all 140 training points
- 48/60random searchthe same budget, spent without a surrogate
One tick is one test point, 0.0167 of the axis, and random search finishes exactly 1 tick ahead of Bayesian optimisation. The untouched configuration sits 0.150 below guessing the majority class; what fixed it was a learning rate 10,000 times larger, not a better model.
Derive
Backpropagation is worked out by hand and implemented from the derivation rather than from a library.
Check
So the gradients are compared against central differences, and then against autograd.
Baseline
The untouched configuration is scored beside the accuracy of simply guessing the majority class - and loses to it.
Tune
What fixed it was the learning rate, four orders of magnitude up, rather than anything about the model.
Resolve
The search with the surrogate loses, by a single correctly classified point out of sixty.
What this does not show
Sixty held-out points, single runs, no seeds repeated. Every margin here is an observation rather than an estimate, and the closest of them is one test point wide - which is why the figure is ruled in test points rather than in percentages. This is coursework: the questions were set rather than chosen, and the problem sheets in the repository are the chair's material, included for context only.
How it was built
System
Workflow and decisions
3 assignments, each with its notebook, its report and the LaTeX source behind it: multiple linear regression and model diagnostics; PCA, logistic regression, SVMs and tree ensembles; backpropagation by hand, Bayesian optimisation, sequence forecasting and Markov decision processes. Figures and tables are generated rather than pasted, and the third assignment carries the provenance ledger this page is built on.
- 1Derive the gradients by hand and implement them
- 2Check them against central differences, then against autograd
- 3Run the default configuration against a majority-class baseline
- 4Sweep the learning rate, then tune with two different searches
- 5Write every figure, table and quoted number from the notebook itself
- Python
- NumPy
- PyTorch
- scikit-learn
- Jupyter
- LaTeX
Evidence
What is actually versioned
Both searches were given the same budget. Bayesian optimisation finished on 47 of 60 held-out points and random search on 48 - a margin of exactly 1 correctly classified point, or 0.0167 of the axis. Reporting that as a percentage would make it look like a finding; reporting it as one point out of 60 is what it is.
Out of the box the network scored 23 of 60, while guessing the majority class scores 32. What closed the gap was a learning rate 10,000 times larger, not a different architecture - and the only reason the failure is visible at all is that the baseline was recorded next to it.
2.8e-17 max absolute difference against PyTorch autograd, and 5.1e-9 max relative error against central differences. The finite-difference figure is bounded by its own step size rather than by the implementation, which is why the two differ by eight orders of magnitude; the autograd comparison is float64 round-off and nothing else.
How the work is checked
Quality controls
How the work is checked
- Every figure and table under the results folders is written by the notebook, so nothing in the reports is transcribed by hand
- The final assignment ships a ledger of 126 reported values across 11 problems, each naming the cell that produced it
- The hand-derived gradients are checked against two independent references rather than one, so a shared mistake has two chances to be caught
- A majority-class baseline is reported beside the model's accuracy, which is what makes the first run legible as a failure rather than as a low number
- The matriculation number is removed from the public copy, and the course's own problem sheets are marked as the chair's material rather than presented as the author's
Limitations
Where the evidence stops
- 60 held-out points. One point is 0.0167 of the accuracy axis, so no difference smaller than that is a difference, and several of the comparisons here are exactly that size
- Single runs rather than repeated ones. There are no seeds-repeated intervals anywhere in this, so the margins reported are observations and not estimates
- Coursework, answering a problem sheet. The questions were set rather than chosen, and the methods are a syllabus rather than a selection made for a problem
- The two hyperparameter searches ran once each on one small problem; nothing here generalises to how either behaves at scale
What this changed in my practice
What this changed in my practice
The ledger changed what I was willing to write. Once each number in the report had to name the cell it came from, the run where the network lost to guessing the majority class had nowhere to go, and it turned out to be the most informative line in the assignment. Reproducibility is usually sold as a courtesy to the reader; in practice it is a constraint on the author.