A working MRM programme for insurance machine learning: what to validate before go-live, how to catch drift in production, and the documentation regulators increasingly expect to see.
Model risk management is the discipline of assuming your models are wrong in ways you have not found yet, and building the process that finds out before your customers or your regulator do. Banking formalised it long ago — the US Federal Reserve's SR 11-7 guidance, published in 2011, remains the reference text — and insurance ML teams inherit a version of the same expectations, increasingly written into law: the EU AI Act's Annex III classifies AI used for risk assessment and pricing of natural persons in life and health insurance as high-risk, with the documentation and oversight duties that follow.
This guide is the practical version: what to build, in what order, sized for an insurance ML team rather than a bank's second line of a hundred people. The core ideas fit in one sentence — know what models you run, validate them independently before they matter, watch them continuously after, and write it all down as you go.
Start with the inventory, because you cannot govern what you cannot list
Every MRM failure story begins the same way: a model in production that nobody owns. Before any validation framework, build the model inventory — a live register, not a spreadsheet someone updates annually.
- Every model in production or feeding production, including vendor models, pricing tables derived from models, and the LLM someone wired into a workflow last quarter.
- A named owner, a named validator, and the decision each model influences — quote, price, referral, claim triage, fraud flag.
- A materiality tier. A model that sets premiums for thousands of policies is not governed like one that orders a work queue. Tiering is what makes the rest of the programme affordable: deep validation for high-materiality models, lighter proportionate review below.
If your inventory conversation surfaces models nobody remembered deploying, that is the programme working. It usually does.
Validation: independent, adversarial, and done before go-live
The single most important word in SR 11-7 is effective challenge — review by people with the skill and the standing to say no, who did not build the model. In a small organisation nobody has a validation department; independence can still be real. The engineer who built the claims model validates the pricing model and vice versa, with a written standard both work to.
What a proportionate pre-deployment validation actually covers.
- Conceptual soundness. Is this technique defensible for this decision, and were obvious alternatives compared? A gradient-boosted model beating a GLM by a hair may not justify its explainability cost on a rating decision.
- Data lineage. Where does every feature come from, what is its refresh cadence, and what happens to the model when the source is late, null, or wrong? Most production incidents are data incidents.
- Performance where it matters. Aggregate accuracy hides the failures that hurt. Evaluate by segment: new business vs renewal, by product, by region, by data completeness. A model that is excellent on average and poor on a segment you are about to grow into is a trap with good metrics.
- Stability and sensitivity. Perturb inputs plausibly; confirm outputs move sensibly. A price that swings on a feature the business considers minor is a conversation to have before launch, not after a complaint.
- Limits of use. Write down what the model was not validated for. The claims triage model repurposed for fraud scoring without revalidation is one of the classic MRM incidents, and the defence is a documented scope someone must consciously exceed.
Drift: production is where models go to age
A validated model starts expiring on deployment day. Portfolios shift, competitors reprice, fraud adapts, a partner integration changes a field's meaning. Drift monitoring is the continuous version of validation, and it needs three layers because each catches what the others miss.
- Input drift. Distribution monitoring on features — population stability metrics, null-rate tracking, new-category detection. Cheap, fast, and catches upstream breakage within hours rather than quarters.
- Output drift. Score distributions, approval rates, average prices by segment. A pricing model whose quotes trend down for a month is telling you something even while every individual quote looks reasonable.
- Outcome drift. Loss ratios, claim frequencies, fraud hit rates against prediction — the ground truth, arriving slowly. Insurance outcomes lag by months, which is exactly why the first two layers exist: they are your early warning while the truth is still in the post.
Every monitored metric needs a threshold and a named response, agreed in calm times. An alert without an owner and a playbook is decoration. Responses scale from investigate, through recalibrate or retrain, to the one teams avoid planning for: fall back. Every material model needs a documented degraded mode — last season's model, a simpler rule set, or manual referral — that can be switched on in hours.
Documentation: written for the reader who was not there
MRM documentation has a specific audience: the regulator, auditor, or successor engineer who arrives in three years knowing nothing. The test of every document is whether that person can understand what the model does, why it was accepted, and what has happened to it since — without interviewing anyone.
The minimum set per material model: a development report (data, method, alternatives considered, known weaknesses stated honestly), an independent validation report with an explicit approve, approve-with-conditions, or reject outcome, the monitoring specification with thresholds and responses, and a change log that records every retrain, recalibration and incident. Honesty about weaknesses is not optional decoration — a development report that lists no limitations is the first thing an experienced reviewer distrusts.
A note on scale and on LLMs
Two honest caveats. First, everything above scales with materiality — a three-person team running one triage model needs the inventory, one good validation, basic monitoring and a one-page fallback plan, not a governance committee. The discipline is proportionality, not paperwork volume.
Second, generative models strain classic MRM at exactly one joint: unbounded outputs make exhaustive pre-deployment validation impossible. The framework still applies, but the emphasis shifts — tighter scoping of what the system may do, evaluation suites run continuously rather than once, and output-side controls doing work that input-side validation cannot. Treating an LLM workflow as exempt from the inventory because it is not a model in the classical sense is how the next generation of unowned production systems is being created right now.
Model risk management has a reputation as the department of slowing things down. Run well, it is the opposite: the teams that can show their models are inventoried, validated and watched are the ones regulators let move quickly, because trust, once documented, is reusable.