Time series on a generated signal
What the Network Adds Over a Calendar
A river flow forecast, measured honestly.
- Role
- Project author
- Classification
- Synthetic demonstration
Problem
A high score may just be the calendar.
How it was solved
I compared the LSTM with a simple calendar lookup.
- Rain and flow
- LSTM
- Calendar lookup
Result
+0.0188
R² added over the calendar
- Calendar lookup0.9025
- LSTM0.9213
- Best possible0.9691
Built with
- PyTorch
- NumPy
- pandas
Synthetic demonstration / Time series
On a signal whose composition is known exactly, does a network reading a weather window beat the simplest thing you would try first? It does, by 0.019, and the honest version of the result is that number rather than the headline.
What the network adds0.0188 R2
A gap of 0.019, read against a ceiling that is known
This diagram in words
- Twenty years, seeded source — Fixed seed, so the series is reproducible and its parts are known.NumPy
- Signal composition artifact — How much is structure and how much is independent noise.
- Chronological split gate — Scalers fitted on the training span only.pandas
- Recurrent network process — Reads a window of weather.PyTorch
- Three baselines process — Including a calendar, which reads no weather at all.
- Scored once gate — A reference run is committed, so a second attempt would show.
- Gap to the ceiling artifact — An axis from nothing to the limit the noise imposes, not truncated.Matplotlib
Read the full background
Problem
Why this work exists
A forecasting score on its own says nothing, because it does not say what was available to be scored. On this signal the two seasonal terms carry 77 per cent of the target's variance and the rain-driven term carries 5.5, so a model can look strong by learning the calendar, and looking strong that way is not evidence of rainfall-runoff skill.
Contribution
What I can claim
Generated the catchment so the components could be measured rather than guessed, computed the ceiling that the generator's own independent noise imposes, and scored the network against three baselines on exactly the same held-out rows, including a lookup table that uses no weather input at all.
Method
Measuring a model against what is possible
A network scores well on a river series.
The gap between a table of 365 numbers and the network is 0.0188. That is the whole of what the network adds, on a catchment that was generated rather than observed.
Generate
Twenty years of discharge from a fixed seed.
Decompose
Most of it turns out to be seasonal, and some of it is independent noise.
Baseline
A lookup table of mean discharge per calendar day is fitted alongside the network.
Score
Both are scored once, after a chronological split with scalers fitted on the training side only.
Concede
The network wins, and the margin over the calendar is small against the distance to the ceiling.
What this does not show
The catchment is generated, not observed: there is no gauge and no site, and nothing here is flood forecasting, a warning system or validated environmental modelling. The ceiling is knowable only because the noise was put in deliberately, which is exactly the circumstance real hydrology never offers.
How it was built
System
Workflow and decisions
A 2-layer LSTM, 128 hidden, dense head of 208,513 parameters reads a 30-day window of three features. The generator, the baselines and the scoring are separate modules, so the thing being measured and the thing it is measured against cannot share an assumption by accident.
- 1Generate twenty years from a fixed seed
- 2Measure what the signal is made of
- 3Split chronologically, fit scalers on train
- 4Score the network and three baselines once
- 5Read the gap against the noise ceiling
- PyTorch
- NumPy
- pandas
- Matplotlib
- pytest
Evidence
What is actually versioned
The network reaches 0.9213 and a lookup table of mean discharge per calendar day reaches 0.9025, on 1,065 held-out days scored once. That gap is the entire value the network adds over 365 numbers.
The generator adds independent noise every day that no method can predict, which caps the achievable score. Reporting a result without it would leave the reader unable to tell a good score from a nearly perfect one.
Against 5.5 per cent for the rain-driven term and 3.1 per cent irreducible noise. This matters more than any score here, because it is what a strong-looking number on this data would mostly be made of.
Mean error above the event threshold, against 0.51 below it. The largest flows are systematically under-predicted, which is ordinary regression to the mean and matters here because the largest flows are the only ones anyone would care about. Recall 0.6852 sits below precision 0.8043 for the same reason.
How the work is checked
Quality controls
How the work is checked
- The ceiling is computed from the generator rather than assumed, so the score can be read against what was achievable
- Three baselines are scored on exactly the same held-out rows, including one that uses no weather input
- Splits are chronological and the scalers are fitted on training rows only; each split builds its windows from its own slice, so none spans a boundary
- The target day is never inside its own input window and discharge is not among the inputs, so the model never sees past discharge
- Persistence is reported but labelled as not a fair comparison, because it uses an input the model does not have
Limitations
Where the evidence stops
- The catchment is generated, not observed. There is no gauge and no site, and nothing here is flood forecasting, a warning system, or validated environmental modelling
- Most of what the model scores is the seasonal cycle, which a table of 365 numbers also learns
- The largest flows are under-predicted, so the regime the work would exist for is the regime it is weakest in
- One generator, one seed, one architecture. Nothing here says the result would hold on a different synthetic catchment, let alone a real one
What this changed in my practice
What this changed in my practice
The generator had a defect worth more than the model did. Its soil-moisture bucket was mis-scaled and sat on its lower bound on 82 per cent of days, so the term it was meant to drive barely moved and the rain-driven share of the target was 1.7 per cent. Rebalanced, it sits at the floor on 13 per cent and the rain share is 5.5. Every result was taken after that, and finding it needed the components to be measurable at all.