MZQMohd Zamin Quadri
All work

Scientific computing demonstration

Synthetic Streamflow Forecasting Benchmark

A fair river flow benchmark on synthetic data.

Role
Project author
Classification
Synthetic demonstration
Inspect repository
R2 0.979 in the repository's earlier run, one step ahead with yesterday's measurement supplied. The two lines overlap because of what the model was given; the current run shows simply repeating yesterday's value scores higher.

  1. Problem

    Two models were ranked on different targets.

  2. How it was solved

    Every method now faces the same days and the same baseline.

    1. Same days
    2. Persistence
    3. Models

  3. Result

    0.994

    ridge, the only method to beat persistence

    • XGBoost0.985
    • Persistence0.988
    • Ridge0.994

  4. Built with

    • XGBoost
    • statsmodels
    • scikit-learn
    See the evidence on GitHub

Scientific computing / Benchmark

A leaderboard invites one reading: rank the rows. These three were not scored on the same task, and the gap between the questions is wider than the gap between the models.

Persistence baselineR2 0.988

A leaderboard whose rows were asked different questions

the same inputsthe horizon was the questionSynthetic seriesFifteen years from a seed. Nota measured river.Chronological holdoutSplit by date, and it decideswhat each row is told.Five-seed repeatThe same fit five times, soone seed cannot be the result.Daily baselinesPersistence and seasonalnaive, given the same…Same task?One step ahead withyesterday's value supplied is…Error analysisWhere the gap between thequestions exceeds the gap…
The baselines are given exactly what the model is given, which is what makes the comparison a comparison. The return path is the finding: what the split hands forward decides the ranking more than the models do.

This diagram in words

  1. Synthetic series source — Fifteen years from a seed. Not a measured river.
    NumPy
  2. Chronological holdout gate — Split by date, and it decides what each row is told.
  3. Five-seed repeat process — The same fit five times, so one seed cannot be the result.
    XGBoost
  4. Daily baselines process — Persistence and seasonal naive, given the same information.
    statsmodels
  5. Same task? gate — One step ahead with yesterday's value supplied is a different question.
  6. Error analysis artifact — Where the gap between the questions exceeds the gap between the models.
    Matplotlib

01 / 13Three numbersA benchmark result, in the shape everyone reads first.

  1. Ridge on featuresR2 0.9938, RMSE 2.0033, scored on 730 daily values
  2. PersistenceR2 0.9882, RMSE 2.7569, scored on 730 daily values
  3. XGBoostR2 0.9848, RMSE 3.1233, scored on 730 daily values
  4. Day-of-year meanR2 -11.692, RMSE 90.2985, scored on 730 daily values
  5. SARIMAX (Monthly)R2 0.963, RMSE 4.679, scored on 24 monthly means
Reported result, and what each row was scored on
  1. Ridge on featuresR2 0.9938RMSE 2.0033, MAE 1.5958, scored on 730 daily values
  2. PersistenceR2 0.9882RMSE 2.7569, MAE 2.139, scored on 730 daily values
  3. XGBoostR2 0.9848RMSE 3.1233, MAE 2.4303, scored on 730 daily values
  4. Day-of-year meanR2 -11.692RMSE 90.2985, MAE 89.9354, scored on 730 daily values
  5. SARIMAX (Monthly)R2 0.963RMSE 4.679, MAE 3.8286, scored on 24 monthly means

Read as a ranking, the table says gradient boosting beats classical statistics by a wide margin. Read against the code, the three rows are not answering the same question. XGBoost is scored one step ahead with the previous day’s measured discharge supplied as an input feature; SARIMAX forecasts 24 monthly means without seeing any of them; the baseline repeats a seasonal average that has no trend term at all.

Q120 days
Observed (white) and predicted (teal) over the first 120 test days. They overlap because the model is predicting tomorrow having been told today.

Two things explain the table. The series was written rather than measured — a fixed-seed generator whose recursion is y[t] = 0.85*y[t-1] + ..., so its day-to-day autocorrelation is 0.9941 and the two most recent lags carry 94.4% of the fitted model while rainfall carries 0.0007. And the baseline is undefended against trend: the same generator adds 0.005 a day, which its own feedback multiplies by 6.67, so the series climbs 89.91 between the training and test periods while the baseline’s mean bias is 89.94. Its MAE (89.9354) and RMSE (90.2985) are nearly equal because the error is an offset, not scatter.

Where it is worst is measurable rather than impressionistic. On the highest tenth of test days — observed flow above 266.28 — the mean absolute error is 5.591 against 2.562 everywhere else, and the signed mean is -5.506: the misses are undershoots. Those days are 10% of the holdout and carry 29% of the squared error. The worst single day observed 273.12 and came in 12.99 low.

Record
5,475 days
Holdout
730 days
Features
26
Horizon
One step
  • Synthetic data, not a catchmentThe river was written by a 40-line generator with a fixed seed. Nothing here is evidence about any real basin, and the repository's own scope note says so first.
  • One step ahead, with the answer in the featuresThe XGBoost score is next-day prediction given discharge already observed at prior timestamps. It is not a recursive multi-day forecast and was never evaluated as one.
  • The three rows are not the same taskThe naive baseline and XGBoost are scored on 730 daily values; SARIMAX is scored on 25 monthly means it forecast without seeing any of them. A single table makes them look comparable.
  • The baseline is undefended against trendA day-of-year mean has no trend term, and the generator injects one. Most of the gap between the baseline and everything else is that, not model quality.
  • No uncertainty of any kindEvery prediction here is a point. The benchmark produces no intervals, no coverage and no calibration, so none is shown.
  • One seed, one runThe tracked figures and JSON record a single reference run. The README notes that library versions may shift the numbers slightly.
Read the full background

Problem

Why this work exists

An earlier version of this repository ranked XGBoost at R2 0.979 above SARIMAX at 0.721. The two numbers were scored on different targets, daily and monthly, and the only baseline offered was a day-of-year mean that could not win. The question became which methods beat a fair baseline on the same rows.

Contribution

What I can claim

Rebuilt the comparison so every daily method is scored on the same 730 held-out days, added persistence, random walk with drift and a trend-corrected day-of-year baseline, added a ridge model on the same features, and repeated the whole comparison over five seeds.

Method

Three numbers that are not the same measurement

A benchmark table invites one reading: rank the rows.

  1. Rank

    Read as a leaderboard, gradient boosting beats classical statistics by a wide margin and the naive baseline is not close.

  2. Open

    One row predicts tomorrow having been given today's measurement.

  3. Attribute

    The two most recent lag features carry most of the strongest model, on a series whose day-to-day autocorrelation is very close to one - because the series was generated by a recursion that is public.

  4. What this does not show

    This benchmark is deterministic and synthetic, and its evaluation is one step ahead with observed discharge supplied as an input. It exercises an evaluation pipeline honestly; it is not a claim about any particular river.

How it was built

System

Workflow and decisions

  1. 1Synthetic 15-year series
  2. 2Chronological holdout
  3. 3Fair daily baselines
  4. 4Five-seed repeat
  5. 5Error analysis
  • XGBoost
  • statsmodels
  • scikit-learn
  • pandas
  • NumPy
  • Matplotlib

Evidence

What is actually versioned

Persistence baselineR2 0.988

Predicting yesterday's value, on the same 730 held-out days as every daily model.

XGBoostR2 0.985

Twenty engineered lag, rolling and calendar features, and still below persistence; it fails to beat it on all five seeds.

Ridge on the same featuresR2 0.994

The only method that improves on persistence, and it does so on all five seeds.

How the work is checked

Quality controls

How the work is checked

  • Fixed-seed data generator and chronological test split
  • Every daily method scored on the same held-out rows
  • Comparison repeated over five seeds and recorded in results/benchmark.json
  • SARIMAX reported on monthly means and kept out of the daily ranking

Limitations

Where the evidence stops

  • Synthetic data cannot establish real-catchment performance
  • The feature models see same-day observed weather, so they are nowcasts rather than pure one-step-ahead forecasts
  • The benchmark does not test recursive multi-day behaviour or distribution shift
What this changed in my practice

What this changed in my practice

A leaderboard is only as good as its weakest baseline. On a series this persistent, repeating yesterday's value is the number every model has to beat.
Source and documentationDiscuss relevant work