Master's thesis on reliable machine learning
Uncertainty Quantification for Traffic Prediction
A traffic model that says how sure it is.
- Role
- Researcher and thesis author
- Classification
- Academic research
- Institution
- Technical University of Munich
Problem
A fast traffic model can be wrong with no warning.
How it was solved
I measured its uncertainty and set aside the answers it trusted least.
- Simulation
- Graph network
- Uncertainty
- Review
Result
41.2%
lower error on the answers it kept
- All predictionsMAE 3.95
- Most confident halfMAE 2.32
Built with
- PyTorch
- PyTorch Geometric
- NumPy
- scikit-learn
Master's thesis / Transport
A graph network answers in milliseconds what a traffic simulator takes hours to compute. The question is not whether it is fast. It is when it may be believed.
Held-out scope100 scenarios
A fast answer, and the policy that decides whether to give it
This diagram in words
- Policy scenarios source — Agent-based simulation runs. Hours each, which is the problem.
- Road-link graph process — Links as nodes. The network is fixed; the demand on it is not.NumPy
- Graph surrogate process — PointNet encoder, TransformerConv and GAT layers.PyTorch · PyTorch Geometric
- Uncertainty estimate process — How unsure the surrogate is, per junction, not per run.
- Review policy gate — Reads the uncertainty and decides whether to answer at all.
- Answered directly sink — Milliseconds instead of hours, where the estimate supports it.Streamlit
- Held for review sink — The predictions the policy declines to make on its own.
Read the full background
Problem
Why this work exists
Paris-scale transport simulations are expensive, while a fast surrogate can fail unevenly across road links and policy regimes. Point predictions alone do not tell an analyst when to trust the approximation.
Contribution
What I can claim
Built on the MATSim corpus and PointNetTransfGAT infrastructure of prior work, then evaluated training variants, MC Dropout, deep ensembles, sigma scaling, split and adaptive conformal prediction, selective prediction, CQR, and error-detection diagnostics.
- 01SimulateCapacity-reduction policy scenarios
- 02RepresentOne graph node per road link
- 03ApproximatePointNet + TransformerConv + GAT
- 04QuantifyDropout, ensembles, and intervals
- 05DecideCalibrate, retain, or route to review
| Stage | Shape |
|---|---|
| PointNet, start position | Local MLP 7 to 256, global MLP 256 to 512 to 512 |
| PointNet, end position | Local MLP 514 to 256, global MLP 256 to 512 to 128 |
| TransformerConv | 128 to 256, four heads |
| TransformerConv | 256 to 512, four heads |
| GATConv | 512 to 64 |
| GATConv | 64 to 1 |
No. A gradient-boosted tree reaches R² 0.7414 against the best graph model’s 0.6841, and trains in about three minutes rather than hours. The graph model earns its place a different way: on ranking its own errors, MC dropout on the weaker network reaches Spearman 0.4817 against the ensemble’s 0.3997, which is what the selective-review result is built on. The tree carries no notion of the network at all.
| Model | R² | MAE (veh/h) | RMSE (veh/h) |
|---|---|---|---|
| XGBoost | 0.7414 | 2.7739 | 5.6933 |
| Deep ensemble | 0.6841 | 3.4853 | 6.2927 |
| Random forest | 0.6612 | 3.2628 | 6.5164 |
| T8 with MC dropout | 0.5856 | 3.9479 | 7.2073 |
| MLP | 0.4928 | 3.8831 | 7.9730 |
| Run | What changed | R² | Split |
|---|---|---|---|
| T1 | First run, Excluded from comparison: a linear final layer, not the graph convolution T2 onward use. | not comparable | 80/15/5 |
| T2 | Correct architecture | 0.5117 | 80/15/5 |
| T3 | Weighted loss, Weighting the loss toward large changes halves the fit. | 0.2246 | 80/15/5 |
| T4 | Weighted loss, dropout back | 0.2426 | 80/15/5 |
| T5 | Smaller batch | 0.5553 | 80/15/5 |
| T6 | Lower learning rate | 0.5223 | 80/15/5 |
| T7 | 80/10/10 split | 0.5471 | 80/10/10 |
| T8 | Lower dropout, The model every uncertainty result on this page is built on. | 0.5957 | 80/10/10 |
| Retained | Accepted-set MAE | Review queue |
|---|---|---|
| 10% | 1.05 veh/h | 2,847,150 |
| 25% | 1.79 veh/h | 2,372,625 |
| 50% | 2.32 veh/h | 1,581,750 |
| 75% | 2.80 veh/h | 790,875 |
| 90% | 3.23 veh/h | 316,350 |
| 100% | 3.95 veh/h | 0 |
Six selected audited operating points from a 5-percentage-point evaluation grid. The interface never interpolates between them and shows no random-review baseline. Rows outside the retained set enter a review queue; the analysis does not label them wrong or unsafe.
Method
How a fast approximation earns the right to be used
An agent-based transport simulation is expensive enough that policy questions get asked less often than they should.
Represent
The network is the input.
Propagate
Information reaches a link by travelling through the structure.
Predict
Every junction now has an estimate.
Quantify
Each junction gets a measure of how much the model had to work with.
Decline
The junctions the model is least sure about are marked rather than hidden, which is what makes a review queue possible instead of a blanket trust decision.
What this does not show
The figure is a construction, not a measurement: ring size is a function of hop distance in this drawing. The measured relationship between uncertainty and error is in the evidence table below, along with what it does not establish.
How it was built
System
Workflow and decisions
The research separates fast surrogate inference from uncertainty estimation, calibration, and the downstream accept-or-review decision. Each layer has a distinct evaluation protocol so confidence claims are not inferred from point accuracy alone.
- 1MATSim policy scenarios
- 2Road-link graph representation
- 3PointNet + TransformerConv + GAT surrogate
- 4Uncertainty estimation
- 5Calibration and review policy
- PyTorch
- PyTorch Geometric
- NumPy
- SciPy
- scikit-learn
- Streamlit
- LaTeX
Evidence
What is actually versioned
3,163,500 road-link predictions in the cached test artifacts.
Deterministic Trial 8; MAE 3.96 veh/h and RMSE 7.12 veh/h.
Pooled Spearman correlation between MC Dropout uncertainty and absolute error.
Accepted-set MAE at 50% retention versus accepting every prediction.
5-member ensemble; MAE 3.49 veh/h and uncertainty-error rho 0.400.
Reported final-thesis split-conformal protocol at nominal 90% / 95%.
Inspection points
Go directly to the evidence
How the work is checked
Quality controls
How the work is checked
- Prediction-to-analysis workflow backed by tracked numeric artifacts
- Aggregate-only local evidence dashboard with automated regression checks
- Calibration protocols versioned instead of combining incompatible splits
- Thesis claims cross-checked against generated reports and cached predictions
Limitations
Where the evidence stops
- One Paris network, one capacity-reduction intervention family, and a fixed 1,000-scenario subset
- Raw MATSim data is not publicly redistributable; a fresh clone cannot reproduce raw simulation-to-graph processing
- Historical preprocessing used split-specific scalers, creating an evaluation-distribution methodology risk
- Uncertainty ranking weakens in some high-change regimes, and coverage is marginal rather than a per-scenario guarantee
- The portfolio application analyses cached artifacts; it is not a live policy simulator
- On the same five features and the same held-out split, a gradient-boosted tree reaches R-squared 0.7414 against the best graph model's 0.6841. The graph surrogate is retained for its uncertainty behaviour, not for its accuracy.
What this changed in my practice
What this changed in my practice
Reliable ML is not one score. Ranking, calibration, conditional behaviour, compute cost, and the operational cost of review must be evaluated together.