Master's thesis / Reliable ML
Reliable GNN Surrogates for Transport Policy Analysis
An evidence-led study of when a graph neural network surrogate is accurate, when it is uncertain, and how that uncertainty can support review decisions.
- Role
- Researcher and thesis author
- Classification
- Academic research
- Institution
- Technical University of Munich
01Problem
Why this work exists
Paris-scale transport simulations are expensive, while a fast surrogate can fail unevenly across road links and policy regimes. Point predictions alone do not tell an analyst when to trust the approximation.
02Contribution
What I can claim
Built on the MATSim corpus and PointNetTransfGAT infrastructure of prior work, then evaluated training variants, MC Dropout, deep ensembles, sigma scaling, split and adaptive conformal prediction, selective prediction, CQR, and error-detection diagnostics.
- 01SimulateCapacity-reduction policy scenarios
- 02RepresentOne graph node per road link
- 03ApproximatePointNet + TransformerConv + GAT
- 04QuantifyDropout, ensembles, and intervals
- 05DecideCalibrate, retain, or route to review
| Stage | Shape |
|---|---|
| PointNet, start position | Local MLP 7 to 256, global MLP 256 to 512 to 512 |
| PointNet, end position | Local MLP 514 to 256, global MLP 256 to 512 to 128 |
| TransformerConv | 128 to 256, four heads |
| TransformerConv | 256 to 512, four heads |
| GATConv | 512 to 64 |
| GATConv | 64 to 1 |
No. A gradient-boosted tree reaches R² 0.7414 against the best graph model’s 0.6841, and trains in about three minutes rather than hours. The graph model earns its place a different way: on ranking its own errors, MC dropout on the weaker network reaches Spearman 0.4817 against the ensemble’s 0.3997, which is what the selective-review result is built on. The tree carries no notion of the network at all.
| Model | R² | MAE (veh/h) | RMSE (veh/h) |
|---|---|---|---|
| XGBoost | 0.7414 | 2.7739 | 5.6933 |
| Deep ensemble | 0.6841 | 3.4853 | 6.2927 |
| Random forest | 0.6612 | 3.2628 | 6.5164 |
| T8 with MC dropout | 0.5856 | 3.9479 | 7.2073 |
| MLP | 0.4928 | 3.8831 | 7.9730 |
| Run | What changed | R² | Split |
|---|---|---|---|
| T1 | First run — Excluded from comparison: a linear final layer, not the graph convolution T2 onward use. | not comparable | 80/15/5 |
| T2 | Correct architecture | 0.5117 | 80/15/5 |
| T3 | Weighted loss — Weighting the loss toward large changes halves the fit. | 0.2246 | 80/15/5 |
| T4 | Weighted loss, dropout back | 0.2426 | 80/15/5 |
| T5 | Smaller batch | 0.5553 | 80/15/5 |
| T6 | Lower learning rate | 0.5223 | 80/15/5 |
| T7 | 80/10/10 split | 0.5471 | 80/10/10 |
| T8 | Lower dropout — The model every uncertainty result on this page is built on. | 0.5957 | 80/10/10 |
| Retained | Accepted-set MAE | Review queue |
|---|---|---|
| 10% | 1.05 veh/h | 2,847,150 |
| 25% | 1.79 veh/h | 2,372,625 |
| 50% | 2.32 veh/h | 1,581,750 |
| 75% | 2.80 veh/h | 790,875 |
| 90% | 3.23 veh/h | 316,350 |
| 100% | 3.95 veh/h | 0 |
Six selected audited operating points from a 5-percentage-point evaluation grid. The interface never interpolates between them and shows no random-review baseline. Rows outside the retained set enter a review queue; the analysis does not label them wrong or unsafe.
Method
How a fast approximation earns the right to be used
An agent-based transport simulation is expensive enough that policy questions get asked less often than they should. A surrogate answers in a fraction of the time - and a surrogate that is confidently wrong is worse than the simulation it replaced. This is the sequence that closes that gap.
Represent
The network is the input. A road link's behaviour depends on what it connects to, so the model reads a graph rather than a table of independent rows.
Propagate
Information reaches a link by travelling through the structure. The wavefront here is a real breadth-first traversal of the fixed network, so the order is the graph's, not a designer's.
Predict
Every junction now has an estimate. On its own that is a number with no indication of when to believe it.
Quantify
Each junction gets a measure of how much the model had to work with. The further information had to travel, the less there was to go on.
Decline
The junctions the model is least sure about are marked rather than hidden, which is what makes a review queue possible instead of a blanket trust decision.
What this does not show
The figure is a construction, not a measurement: ring size is a function of hop distance in this drawing. The measured relationship between uncertainty and error is in the evidence table below, along with what it does not establish.
03System
Workflow and decisions
The research separates fast surrogate inference from uncertainty estimation, calibration, and the downstream accept-or-review decision. Each layer has a distinct evaluation protocol so confidence claims are not inferred from point accuracy alone.
- 01MATSim policy scenarios
- 02Road-link graph representation
- 03PointNet + TransformerConv + GAT surrogate
- 04Uncertainty estimation
- 05Calibration and review policy
- PyTorch
- PyTorch Geometric
- NumPy
- SciPy
- scikit-learn
- Streamlit
- LaTeX
04Evidence
What is actually versioned
3,163,500 road-link predictions in the cached test artifacts.
Deterministic Trial 8; MAE 3.96 veh/h and RMSE 7.12 veh/h.
Pooled Spearman correlation between MC Dropout uncertainty and absolute error.
Accepted-set MAE at 50% retention versus accepting every prediction.
5-member ensemble; MAE 3.49 veh/h and uncertainty-error ρ 0.400.
Reported final-thesis split-conformal protocol at nominal 90% / 95%.
05Inspection points
Go directly to the evidence
06Quality controls
How the work is checked
- Prediction-to-analysis workflow backed by tracked numeric artifacts
- Aggregate-only local evidence dashboard with automated regression checks
- Calibration protocols versioned instead of combining incompatible splits
- Thesis claims cross-checked against generated reports and cached predictions
07Limitations
Where the evidence stops
- One Paris network, one capacity-reduction intervention family, and a fixed 1,000-scenario subset
- Raw MATSim data is not publicly redistributable; a fresh clone cannot reproduce raw simulation-to-graph processing
- Historical preprocessing used split-specific scalers, creating an evaluation-distribution methodology risk
- Uncertainty ranking weakens in some high-change regimes, and coverage is marginal rather than a per-scenario guarantee
- The portfolio application analyzes cached artifacts; it is not a live policy simulator
- On the same five features and the same held-out split, a gradient-boosted tree reaches R-squared 0.7414 against the best graph model's 0.6841. The graph surrogate is retained for its uncertainty behaviour, not for its accuracy.
What this changed in my practice
Reliable ML is not one score. Ranking, calibration, conditional behavior, compute cost, and the operational cost of review must be evaluated together.