Actuarial Model Risk Management: GLM to LLM Validation

Validating an actuarial model means showing the three things bank supervisors named in 2011 and kept in April 2026: conceptual soundness, ongoing monitoring and outcomes analysis (SR 26-2, April 2026; the site's coverage). A rating GLM and a gradient-boosted model can show all three, because each can be back-tested on a period it was not fitted to. An LLM-assisted reserving process has no back-test, and the 2026 guidance leaves generative models out of its scope.

ASOP No. 56, in force since October 1, 2020, tells the actuary to validate model output and to disclose reliance on models built by others. The NAIC's December 2023 model bulletin asks for a written AI systems program. It had been adopted by 25 states and the District of Columbia by July 2026 (Quarles, 2026). Neither says which test satisfies which requirement for which model.

Key Takeaways

Bank supervisors kept the three validation elements of 2011 when the 12-page SR 26-2 replaced the 21-page SR 11-7 on April 17, 2026.
A rating GLM and a GBM pass outcomes analysis with a lift back-test on an out-of-time year; SR 11-7 says holdout samples are not substitutes for it.
An LLM-assisted process has no outcomes test. Its substitute is a documented review rate: 300 sampled outputs with no error found bound the error rate near 1%.
The NAIC's RBC Model Governance Task Force exposed draft model risk guidelines on October 8, 2026, built on ASOP 56 and SR 11-7, with a 3 to 5 model pilot.

The Three Elements on a Rating GLM

SR 11-7 set the definition in April 2011: "An effective validation framework should include three core elements: evaluation of conceptual soundness, including developmental evidence; ongoing monitoring, including process verification and benchmarking; outcomes analysis, including back-testing" (Federal Reserve, April 2011). SR 26-2 kept the three, cut the attachment from 21 pages to 12 and tied the depth of validation to a model's materiality (Federal Reserve, April 2026).

For a personal lines rating GLM, conceptual soundness is the developmental evidence: the variable selection memo, each coefficient against its standard error, and the stability of the relativities when the model is refit on subsets of the training years.

Ongoing monitoring is the factor-level actual-to-expected report, with a tolerance that triggers a refit. SR 11-7 adds a test actuaries rarely file, the override rate: "If the rate of overrides is high, or if the override process consistently improves model performance, it is often a sign that the underlying model needs revision or redevelopment" (SR 11-7, April 2011).

Outcomes analysis is where fit statistics stop counting. SR 11-7 defines back-testing as comparing "actual outcomes with model forecasts during a sample time period not used in model development." It adds that in-sample fit and holdout samples "are not substitutes for back-testing." A lift chart and a double-lift chart against the incumbent plan, run on the accident year after the fitting window closed, is the test. A holdout drawn at random from the same years is not.

The Same Tests on a Gradient-Boosted Model

A GBM changes the evidence in the first two cells. Conceptual soundness rests on partial dependence plots and monotonic constraints on features with a known direction. It also needs a documented answer to ASOP 56 section 3.1.4(d), "whether there is a material risk of the model overfitting the data" (Actuarial Standards Board, 2020).

Monitoring adds a drift measure the GLM did not need. Feature importance recomputed each quarter catches a model whose weight has shifted onto a variable whose distribution has moved. The NAIC bulletin lists "model drift" among the assessments an AI systems program should carry (NAIC, December 2023, section 3.3(c)).

Outcomes analysis is unchanged: the same back-tests on the out-of-time year. ASOP 56 section 3.4 asks the actuary to understand "the designer's or developer's original intended purpose" and to disclose the extent of reliance. Colorado's first annual AI compliance reports, due July 1, 2026, showed the scale of that reliance: 40% of reported models came from outside vendors.

Where the LLM Column Has No Test

An LLM-assisted process, one that drafts the reserve narrative, changes the first two cells and empties the third. Conceptual soundness becomes documentation of the prompt, the retrieval set, the model version and its sampling settings. Monitoring becomes output sampling.

Outcomes analysis has no back-test, because there is no forecast to set against a realized value. SR 26-2 has a line written for a different problem that fits this one: "When a model's design relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment." In the LLM column the expert review is the control itself.

The substitute is a written review rate and an error rate measured on the sample. The NAIC bulletin allows for it: validation can take the form of "measuring performance against expert review, or other methods" (NAIC, December 2023, section 3.4). Under the rule of three, a random sample of 300 outputs with no error found bounds the true error rate at about 1% with 95% confidence. A unit reviewing 5% of 2,000 monthly outputs reads 100 and can claim an error rate under 3% only if it finds none.

The site's review of LLM validation in P&C claims covered the other control in this cell, ensemble sampling: the same prompt run dozens of times and the variance certified. Neither control is an outcomes analysis in the supervisory sense. The April 2026 guidance leaves the column out: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance" (OCC Bulletin 2026-13 and SR 26-2, April 17, 2026).

Requirement (source)	Rating GLM	Gradient-boosted model	LLM-assisted process
Conceptual soundness (SR 11-7, SR 26-2)	Coefficients against standard errors; refit stability; selection memo	Partial dependence; monotonic constraints; overfitting check (ASOP 56 3.1.4)	Prompt, retrieval set, version and settings documented
Ongoing monitoring (SR 11-7, SR 26-2; NAIC 3.3(c))	Factor-level actual-to-expected; override rate	Feature-importance drift; actual-to-expected	Output sampling at a stated rate; version log
Outcomes analysis (SR 11-7, SR 26-2; ASOP 56 3.6.2)	Lift and double-lift back-test on an out-of-time year	Same back-test	None. Substitute: human review rate and sampled error rate
Limitations (ASOP 56 3.2)	Documented scope and weaknesses	Same, plus behaviour outside the training range	Fabrication, prompt sensitivity, non-reproducible output
Reliance on others (ASOP 56 3.4; SR 26-2 VII; NAIC 4.2)	Disclosure if vendor-built	Disclosure; vendor developmental evidence	Disclosure; foundation model beyond audit reach; contract audit rights
Program and inventory (NAIC 3.3(a); supplement Exhibit A)	Inventory entry	Inventory entry	Inventory entry; the supplement names generative AI

How the table was built: rows one to three are the validation elements of SR 11-7 and SR 26-2; rows four to six are requirements of ASOP No. 56, the NAIC model bulletin and the AI Risk Evaluation Supplement. Each cell names the test or control that satisfies the requirement.

Which Cells the Insurance Documents Fill

Read across, the insurance documents fill the lower three rows and leave the top three to the bank text. ASOP 56 fills limitations and reliance. Its own output validation list in section 3.6.2, historical actuals, hold-out data and an alternative model, stops at the holdout SR 11-7 says is not a back-test. The NAIC bulletin fills the program row, and version 5.0 of the AI Risk Evaluation Supplement turns the inventory it asks for into an examination request across 13 program areas.

The top three rows now have an insurance author for one class of model. The NAIC's Risk-Based Capital Model Governance Task Force exposed draft "Aspirational Risk-Based Capital Model Risk Management Guidelines" on October 8, 2026 (NAIC, October 2026). The draft takes its definitions of model and model risk from ASOP 56 and SR 11-7 by name and proposes a pilot of three to five RBC models. It says nothing about generative AI.

The Academy's Cross-Practice RBC Task Force answered on November 2 with a one-page letter pointing the task force to its May 2019 Model Risk Management practice note (American Academy of Actuaries, November 2026). That leaves the LLM column where SR 26-2 left it. The SOA's Foresight Project found senior actuaries describing LLM use as wide and shallow. The supplement that would ask their firms for an inventory comes up for adoption at the NAIC Fall National Meeting.

Model Risk Management for Actuarial Models, From GLM to LLM

Key Takeaways

The Three Elements on a Rating GLM

The Same Tests on a Gradient-Boosted Model

Where the LLM Column Has No Test

Which Cells the Insurance Documents Fill

Further Reading

Sources