Master's thesis submitted
Uncertainty Quantification for Machine Learning Models in Transportation Policy Analysis
A scientific record of what was tested, which reliability property each method measures, what the aggregate evidence supports, and where the result stops.
01Research question
Fast prediction is useful only when failure is visible.
Agent-based transport simulations can be expensive to run. A graph neural network surrogate can approximate the change in traffic volume on each road link, but a point estimate alone cannot tell an analyst where that approximation is likely to fail.
02System context
From simulation output to a graph surrogate
Each scenario becomes a road-link graph. A PointNet, TransformerConv, and GAT model family learns the policy-induced traffic-volume change at each link. Uncertainty is then evaluated as a separate layer rather than inferred from point accuracy.
- 01SimulateCapacity-reduction policy scenarios
- 02RepresentOne graph node per road link
- 03ApproximatePointNet + TransformerConv + GAT
- 04QuantifyDropout, ensembles, and intervals
- 05DecideCalibrate, retain, or route to review
03Methods
Different tools answer different reliability questions.
MC Dropout and deep ensembles estimate spread in different ways. Calibration tests whether that spread corresponds to observed error. Conformal prediction targets empirical marginal coverage. Selective prediction uses a ranking to allocate review; it does not calibrate an interval.
Deterministic GNN
How accurate is the surrogate?
- Mechanism
- One point prediction per road link
- Finding
- Trial 8 reported R² 0.596 and MAE 3.96 veh/h.
- Status
- Historical held-out result
MC Dropout
Does uncertainty rank likely error?
- Mechanism
- Thirty stochastic passes through one model
- Finding
- Uncertainty–absolute-error Spearman ρ 0.482; raw spread remained under-dispersed.
- Status
- Trial-specific cached replay
Deep Ensemble
Do multiple fits improve prediction and spread?
- Mechanism
- Five independently trained model members
- Finding
- R² 0.684 and MAE 3.49 veh/h; uncertainty–error ρ 0.400.
- Status
- Cached full-test recomputation
Conformal prediction
Do intervals reach a stated marginal coverage?
- Mechanism
- Held-out residual quantiles around predictions
- Finding
- 90.02% and 95.01% empirical marginal coverage at nominal 90% and 95%.
- Status
- Submission-era reported result
Accuracy, ranking, spread, calibration, and coverage are different properties. A stronger score in one column does not establish the others.
All evaluated method families
- Deterministic GNN baselines
- MC Dropout
- Deep ensembles
- Regression sigma scaling
- Split and adaptive conformal prediction
- Selective prediction and error-detection diagnostics
- Conformalized quantile regression
04Selective prediction
Choose review capacity; observe the retained-set trade-off.
Trial 8 MC Dropout uncertainty was used to sort predictions from least to most uncertain. The interaction below exposes six audited operating points only. Moving a prediction to review is a routing decision, not a declaration that it is wrong.
- Retained
- 50%
- Accepted-set MAE
- 2.32 veh/h
- Lower than full retention
- 41.2%
- Review queue
- 1,581,750
| Retained | Retained rows | Review queue | MAE |
|---|---|---|---|
| 10% | 316,350 | 2,847,150 | 1.05 veh/h |
| 25% | 790,875 | 2,372,625 | 1.79 veh/h |
| 50% | 1,581,750 | 1,581,750 | 2.32 veh/h |
| 75% | 2,372,625 | 790,875 | 2.80 veh/h |
| 90% | 2,847,150 | 316,350 | 3.23 veh/h |
| 100% | 3,163,500 | 0 | 3.95 veh/h |
Retrospective Trial 8 MC Dropout triage on the cached 100-graph archive. No random-review baseline is shown, and the operating points do not transfer automatically to a new model, network, intervention, or data distribution.
05Calibration & coverage
Ranking error is not the same as calibrating uncertainty.
A model can rank hard cases while still producing under-dispersed uncertainty. Scaling was evaluated under two distinct protocols, and split conformal coverage is reported separately as marginal evidence under exchangeability assumptions.
Graph-level audit protocol
First 20 graphs calibrate / last 80 evaluate
- Before ECE
- 0.269
- After ECE
- 0.048
- Scale
- 2.702
Tracked aggregate result; regeneration requires controlled source artifacts.
Final-thesis node protocol
Random 30% node calibration / 70% node evaluation
- Before ECE
- ≈0.356
- After ECE
- ≈0.034
- Scale
- ≈2.887
Reported result; canonical split indices are unavailable.
ECE is expected calibration error; lower is better. These protocols use different units and splits, so their values are shown as separate records rather than one combined trend.
Observed empirical marginal coverage
Observed empirical marginal coverage
Submission-era reported result from a 50/50 scenario split. Marginal coverage does not guarantee the same coverage for every scenario, road link, uncertainty stratum, or city.
06Findings
What the aggregate evidence supports
5 members produced the strongest cached-test point accuracy: R² 0.684 and MAE 3.49 veh/h.
Trial 8 produced the stronger uncertainty–absolute-error rank association: Spearman ρ 0.482.
Accepted-set MAE was 2.32 veh/h, 41.2% below accepting every cached prediction.
Reported split-conformal coverage was 90.02% and 95.01% at nominal 90% and 95%.
07Reproducibility boundary
Public aggregate evidence, explicit exclusions
The submitted PDF remains immutable and contains a numerical claim corrected after submission. The corrigendum and audited aggregate bundle are the current evidence boundary for numerical claims.
Public record
- Immutable submitted PDF with a separate post-submission corrigendum
- Aggregate JSON, CSV, report, figures, and artifact hashes
- Source code, local dashboard code, validation, and regression tests
Not published here
- Raw MATSim scenarios and graph topology
- Row-level predictions, targets, and uncertainty arrays
- Graph loaders, split-specific scalers, and model checkpoints
- Private dashboards, local paths, and confidential source material
08Scientific limits
What this thesis does not establish
- One Paris network, one capacity-reduction intervention family, and a fixed 1,000-scenario subset
- Raw MATSim data is not publicly redistributable; a fresh clone cannot reproduce raw simulation-to-graph processing
- Historical preprocessing used split-specific scalers, creating an evaluation-distribution methodology risk
- Uncertainty ranking weakens in some high-change regimes, and coverage is marginal rather than a per-scenario guarantee
- The portfolio application analyzes cached artifacts; it is not a live policy simulator
- On the same five features and the same held-out split, a gradient-boosted tree reaches R-squared 0.7414 against the best graph model's 0.6841. The graph surrogate is retained for its uncertainty behaviour, not for its accuracy.
- Pooled road links are dependent within scenarios; they are not independent observations.
- MC Dropout replays are stochastic, and thresholds require recalibration after model or distribution changes.