Insurance claims prediction
A Probability Is Not a Decision
Turning a risk score into a decision.
- Role
- Author
- Classification
- Engineering prototype
Problem
A good ranking is still not a decision.
How it was solved
I split it into rank, calibrate and decide, using real costs.
- Rank
- Calibrate
- Decide
Result
3.23%
the risk where acting pays off
Built with
- scikit-learn
- XGBoost
- LightGBM
- SHAP
Research pipeline / Classification
A claims model gives every policy a number. Someone still has to decide which numbers are worth acting on - and the cost of being wrong is wildly different in the two directions, so where does the line actually belong?
Where the cost matrix puts the cut3.23%
A probability, calibrated, and then turned into a decision
This diagram in words
- Split first gate — Encoders and scaling fitted on the training side only.pandas
- Classifiers compared process — Behind one interface, so this is not a comparison of harnesses.scikit-learn · XGBoost · LightGBM
- Calibrated process — Platt or isotonic, cross-validated, scored by Brier.
- Threshold searched gate — Against the cost criterion. The two costs are wildly unequal.
- Attribution artifact — What the chosen model rests on, computed after it is chosen.SHAP
- Worth acting on sink — Above the line the costs put it, not above one half.
- Left alone sink — Where acting costs more than being wrong about it.
Read the full background
Problem
Why this work exists
A classifier that ranks well can still be badly wrong about magnitude, and a probability that is right is still not an action. Most of the distance between a model and a decision lives in those two gaps, and collapsing them into one accuracy number hides both - along with the fact that in claims the two kinds of error cost wildly different amounts.
Contribution
What I can claim
Built the pipeline as three separable stages rather than one: a train/test split with encoders and scaling fit on training data only, classifiers compared under one interface, cross-validated calibration scored by Brier, and a threshold search that can optimise for statistical or cost-weighted criteria. The cost matrix is a parameter of that search, not a constant buried in it.
Method
Three problems that get reported as one
A claims model is usually judged by a single number, which quietly assumes that ranking policies, stating honest probabilities and deciding what to do about them are the same achievement.
The matrix those four numbers come from is the repository’s own example and is not a validated policy. The crossing is arithmetic, not a result: no data, no model and no score are published here.
Rank
The first question is only whether the model orders policies by risk at all.
Calibrate
The second is whether the numbers mean what they say - whether policies given a one in five chance claim about one time in five.
Weigh
The two errors are not comparable.
Cut
Given calibrated probabilities and those costs, the break-even follows as arithmetic: act when acting is worth more than not acting.
Attribute
Last, the decision has to be explainable to someone who did not build it, which is why attribution is part of the pipeline rather than a postscript to it.
What this does not show
The cost matrix is the repository's own example and is not a validated business policy. Nothing here is a result: no data, split, trained model, calibration output or evaluation report is versioned, so there is no accuracy, no AUC and no business value to report - and the code must not be used to make automated decisions about people or policies.
How it was built
System
Workflow and decisions
Five modules, one per stage: the data pipeline, the classifier comparison across logistic regression, random forest, XGBoost, LightGBM, the calibration fit, the threshold search, and SHAP attribution. Generated artefacts stay out of version control, which is why the repository can be read without being mistaken for a result.
- 1Split, then fit encoders and scaling on the training side only
- 2Compare the classifiers under one interface
- 3Calibrate with Platt scaling or isotonic regression, cross-validated, scored by Brier score
- 4Search the decision threshold against the chosen criterion
- 5Attribute the resulting model with SHAP
- Python
- scikit-learn
- XGBoost
- LightGBM
- SHAP
- pandas
Evidence
What is actually versioned
The repository's example matrix values a claim caught early at +500, an unnecessary investigation at -50, and a missed claim at -1000. Setting the value of acting equal to the value of not acting solves to 3.23% with no dataset involved - 15.5 times below the 0.5 a classifier defaults to. That is arithmetic from four published numbers, and the matrix they come from is illustrative rather than a validated policy.
The repository versions no data, no train/test split, no trained model, no calibration output and no evaluation report, and states that it therefore makes no verified accuracy, AUC, calibration-improvement or business-value claim. This page makes none either.
rank - does the model order policies by risk at all; calibrate - does a stated 20% mean twenty in a hundred; cut - where the probability becomes an action. The cut can be chosen 4 ways: best F1, Youden's J, most expected value, precision at 90% recall.
How the work is checked
Quality controls
How the work is checked
- Encoders and scaling are fit on the training split only, so the test side is not quietly informed by the data it is meant to judge
- Calibration is cross-validated and scored by Brier rather than asserted, because a calibration step that is not measured is a hope
- The cost matrix is an argument to the threshold search rather than a constant inside it, which is what makes it reviewable
- The README states the matrix is illustrative and must not be used without domain, legal, fairness and risk review, rather than leaving a reader to assume it was validated
Limitations
Where the evidence stops
- No data, no split, no trained model, no calibration output and no evaluation report are versioned. There is no performance claim here because there is nothing to substantiate one
- The cost matrix is an example. The threshold derived from it is correct arithmetic about those four numbers and says nothing about what a real insurer's numbers would be
- Calibration method and threshold should be chosen on a validation set kept separate from the final test set; the supplied scripts are exploratory and do not implement that governance
- It must not be used to make automated decisions about people or policies, which is the repository's own instruction and the reason this page stops at the arithmetic
What this changed in my practice
What this changed in my practice
The threshold is where the modelling stops and the policy starts, and it is the step most likely to be left at a default nobody chose. Working the cost matrix through to a break-even was the moment the project became about decisions rather than about classifiers: the honest answer was fifteen times away from the habit, and it came from four numbers rather than from any amount of model tuning.