Forecasting
A Published Margin That Does Not Survive an Ordinary Year
Forecasting Munich's road accidents.
- Role
- Project author
- Classification
- Reproducible experiment
Problem
The baseline was forecasting a pandemic year.
How it was solved
I reran the same test on ordinary years.
- Published window
- Ordinary years
- Compare
Result
+23.4% → −15.5%
the model's margin in ordinary years
- Published window+23.4%
- Ordinary years−15.5%
Built with
- scikit-learn
- pandas
- FastAPI
- pytest
Reproducible experiment / Forecasting
The model beats its baseline by 23.4 per cent. Run the same protocol three years earlier and the sign flips, because the baseline on the published window repeats a pandemic.
The published margin+23.4%
The same protocol, run on two windows
This diagram in words
- Seven published series source — City open data, one model fitted per series.pandas
- Trend window chosen gate — Selected on two validation years, before anything is scored.
- Refit and score once process — On the two years after the ones that chose the window.scikit-learn
- Published window process — The window the original 23.4% margin was measured on.
- Pre-pandemic window process — The identical protocol, three years earlier.
- Bootstrap both margins gate — Over held-out months, and applied to both runs rather than one.pytest
- Both margins at zero artifact — One stays clear of zero. The other crosses it, and that is the finding.
Read the full background
Problem
Why this work exists
The model beats a seasonal-naive baseline by 23.4% on the published window, on all seven series. That baseline repeats the previous calendar month, and on this window the previous year is 2020, so the baseline forecasts a pandemic. Saying so in a caveat is not the same as measuring it.
Contribution
What I can claim
Ran the identical protocol three years earlier, over a stretch where nothing unusual happened, with the same code, the same metrics and the same number of held-out rows. Then measured both margins against resampling, because a single number on 24 months says nothing about whether it would hold on a different 24 months.
Method
The same protocol, asked twice
A forecasting model beats its baseline on a published series, and the margin looks worth reporting.
The lower interval crosses zero, so on 24 months that margin is not separable from no difference at all.
Model
One model per published series, each fitted on its own history.
Select
The trend window is chosen on validation years and then left alone.
Score
Refit once and score once on the years that follow.
Repeat
The identical protocol is then re-run on an earlier, ordinary window.
Bound
Both margins are bootstrapped over held-out months, and neither is comfortably away from zero.
What this does not show
Both margins rest on twenty-four months across seven series. Neither should be read as settled, and the pre-pandemic one explicitly cannot be distinguished from zero. The series are published counts of recorded accidents, which is not the same as accidents.
How it was built
System
Workflow and decisions
One protocol, parameterised by which years are validation, which year training ends on, and which two years are held out. The published run and the pre-pandemic run are two calls to it, which is what makes them comparable at all.
- 1Seven published series, one model each
- 2Select the trend window on two validation years
- 3Refit and score once on the two years after
- 4Re-run the whole protocol on a pre-pandemic window
- 5Bootstrap both margins over held-out months
- scikit-learn
- pandas
- FastAPI
- pytest
- GitHub Actions
Evidence
What is actually versioned
Scaled MAE 0.1531 against a seasonal-naive 0.1999 on 168 held-out monthly observations, winning on 7 of 7 series. A 3-month moving-block bootstrap puts it between 7.2 and 33.3 per cent, which stays clear of zero.
Selected on 2016 and 2017, scored once on 2018 to 2019. The sign flips and the model wins on 3 of 7 series. Its interval runs -36.8 to 6.9 per cent, so on 24 months this margin cannot be told apart from no difference either way.
Moving to an ordinary period costs more than the entire published margin, which is what the repository records rather than what it claims for the method.
7 series, 2000 to 2022, from the Landeshauptstadt Muenchen, open data portal. Errors are scaled by each series' own level before averaging, because the largest series is about 169 times the smallest.
How the work is checked
Quality controls
How the work is checked
- Both windows are produced by the same function, so the two rows are not two scripts that happen to agree
- The held-out years are scored once; the trend window is chosen on separate validation years
- The bootstrap is paired, so every method is scored on the same resampled months and what varies is the margin rather than the noise in each
- It is run twice, independently and in blocks of three months, and the wider interval is the one the README tells you to believe
- Errors are scaled per series before averaging, so the small series are not drowned out by the large ones
Limitations
Where the evidence stops
- Both margins rest on 24 months of seven series. Neither row should be read as settled, and the pre-pandemic one explicitly cannot be distinguished from zero
- The two windows also select different models: a 12-year trend window against a 6-year one. Same procedure, different data, so the comparison is between two protocol runs rather than one model on two periods
- Counts only. No exposure denominator, no severity weighting, and no causal claim about why any series moves
- A coding-challenge project rebuilt afterwards, not a deployed forecasting service
What this changed in my practice
What this changed in my practice
A baseline carries its own history, and on this window the history was a pandemic. The honest way to find that out was not to add a sentence of caveat but to run the whole protocol somewhere the pandemic was not, and then to check whether either answer survives being resampled. Only one of them does.