NAIC's AI Evaluation Tool Turns Bulletins Into a Scored Exam

The NAIC's AI Systems Evaluation Tool, a four-exhibit examiner framework now piloting across 12 states through September 2026, converts the 2023 Model Bulletin's disclosure language into a scored market conduct exam instrument. Adoption is expected at the NAIC's November 2026 Fall National Meeting (Fenwick, June 2026), and that vote, not the pilot itself, is the deadline actuarial teams should be building toward.

From tracking every NAIC Big Data and Artificial Intelligence Working Group exposure draft since the Model Bulletin's December 2023 debut, the shift from voluntary disclosure to a four-exhibit exam instrument that pilot-state examiners can actually score is the clearest signal yet that AI governance has moved from a compliance memo to a line item examiners will test. Ten insurance companies are enrolled across the 12 pilot states (InsuranceNewsNet, citing Iowa Insurance Commissioner Doug Ommen, 2026), and what those companies are being asked to produce, particularly under Exhibit D's data integrity standard, has no equivalent in current rate-filing documentation. Most coverage of the pilot has treated it as another compliance headline. What has not been walked through mechanically is what Exhibit D actually demands of a working rating engine, and that gap is where actuarial teams are most exposed.

The Four Exhibits, and What Each One Demands as Evidence

The Evaluation Tool is built to let an examiner escalate scrutiny in stages rather than open every carrier to the same depth of inquiry. Exhibit A asks a company to quantify its AI footprint: how many systems are in production, by function, and which ones touch a consumer-facing or financially material decision. Exhibit B is a governance risk assessment framework, offered in narrative or checklist form, covering roles, oversight, and how AI risk feeds into enterprise risk management. Exhibit C narrows to the systems the company itself has classified as high-risk, requiring development history, testing results, and human-in-the-loop detail. Exhibit D is the data exhibit: sourcing, quality controls, representativeness, and a new field the pilot draft added for "Reasonable Accommodations or Policy Modifications" (Monitaur, 2026).

Each exhibit maps to a distinct evidentiary burden, and the mapping is where the tool differs sharply from the bulletin it is built on. The bulletin asked a carrier to describe its governance program in a filing narrative. The exhibits ask for the underlying artifacts an examiner can cross-check against the narrative, which is a different kind of documentation discipline entirely.

Exhibit	What it asks for	What the actuarial/model-risk team must produce
A: AI usage inventory	Count and classify every production AI system by function and impact	A queryable model inventory tying each model to owner, use case, and consumer/financial materiality
B: Governance risk framework	Narrative or checklist description of oversight, vendor management, ERM integration	Documented model risk policy, escalation paths, and a record of how AI risk is reported into ORSA
C: High-risk system detail	Development, testing, and human-in-the-loop detail for self-classified high-risk models	Written high-risk classification criteria, applied consistently, plus validation and bias-testing records
D: Data integrity	Source, lineage, quality checks, and representativeness of training and input data	A traceable lineage from raw source through feature construction to the model's production inputs

The proportionality language in the pilot materials tells examiners to spend more time on Exhibit C and D scrutiny for systems that could cause serious consumer or financial harm, and less on low-risk back-office automation (Monitaur, 2026). That principle sounds like restraint. In practice it concentrates the heaviest documentation burden precisely on pricing and underwriting models, the systems most likely to be scored as high-risk under any reasonable reading of the exhibits.

How the Scoring Logic Compares to Existing Exam Frameworks

Financial condition exams have run on a risk-focused surveillance framework for two decades: examiners triage a carrier's branches and functions by inherent risk, then calibrate testing depth to what that triage reveals. Market conduct exams have historically worked from a narrower complaint-and-sampling model, pulling files after a trigger rather than scoring an entire operational category up front. The Evaluation Tool imports the financial-exam logic of risk-tiered triage into the market conduct space, and it does so through a four-tier severity scale, from unacceptable down through high, medium, and low, that regulators floated at the NAIC's Spring 2026 National Meeting (Mayer Brown, April 2026). That scale echoes the EU AI Act's own four-tier risk classification, though the NAIC version stays inside existing state exam authority rather than creating new statutory categories.

What counts as "high-risk" is, for now, largely self-determined. The current draft lets a company set its own high-risk criteria in Exhibit C, which gives carriers room to build a defensible methodology but also invites an examiner to challenge the boundary once the tool graduates from pilot to adopted instrument. Regulators have signaled that high-risk is not limited to systems with direct consumer contact; models affecting solvency or financial risk fall inside the same lens (Monitaur, 2026). For a pricing actuary, that reach is the detail worth sitting with: a rating factor engine used purely for internal loss-cost estimation, with no consumer-facing output, can still land in the high-risk tier on a material-financial-impact reading of the exhibits.

Twelve Exam-Ready States Against a 24-State Disclosure Base

The compliance gap the pilot creates is easiest to see in the state count. As of April 2026, 24 states and the District of Columbia had adopted the Model Bulletin, with four more states applying comparable guidance outside formal bulletin adoption (Quarles Law Firm, April 2026). The 12-state pilot footprint, California, Colorado, Connecticut, Florida, Iowa, Louisiana, Maryland, Pennsylvania, Rhode Island, Vermont, Virginia, and Wisconsin, sits inside that base: exactly half of the jurisdictions that have adopted bulletin-level disclosure expectations are now testing whether they can turn those expectations into a scored exam (NAIC Pilot Project Summary, 2026).

That 50% overlap is the practical signal for a multi-state carrier. A company operating in all 24-plus bulletin states has, until now, faced a fairly uniform disclosure obligation: describe the governance program, attest to bias testing, respond to complaints. In the 12 pilot states, that same carrier may now face an examiner walking through Exhibit C and D with a specific model in hand, asking for lineage documentation the bulletin never required in writing. The other 12-plus bulletin-only states remain, for now, in the lighter disclosure regime. That asymmetry is temporary by design. The pilot's stated purpose is to produce a tool the Working Group can re-expose in September and October 2026 before a possible adoption vote at the November 2026 Fall National Meeting, at which point the exam-ready standard, not the disclosure-only one, becomes the baseline every bulletin-adopting state is expected to reach for (Fenwick, June 2026).

What Exhibit D Would Actually Demand of a Rating Engine

Take a concrete case: a personal auto insurer running a gradient-boosted model to derive rating factor relativities, feeding a GLM-based base rate structure that gets filed with state regulators. Under current practice, the actuarial documentation supporting that filing centers on the rate filing memorandum itself, built to satisfy ASOP No. 12 (risk classification) and ASOP No. 56 (modeling), plus whatever internal model validation the carrier's governance policy requires. That documentation explains what the model does and why its outputs are actuarially sound. It does not, in most shops, trace the full data lineage from raw source file through feature engineering to the specific values that entered model training.

Exhibit D asks for exactly that trace. It wants the sources feeding the model, internal policy and claims data alongside any third-party or purchased data, the quality controls applied at each stage, and an assessment of whether the training population is representative of the population the model will be applied to. For a gradient-boosted rating engine with dozens of candidate features, several of which may originate from licensed third-party data providers (credit-based insurance scores, motor vehicle records, property attribute vendors), producing that lineage means the actuarial team has to be able to answer, for every feature in the final model, where the raw value came from, what transformation produced the feature, and what quality check confirmed the transformation ran correctly. A rate filing memorandum answers "is this factor actuarially justified." Exhibit D answers a different question: "can you prove, end to end, that the data behind this factor is what you say it is."

Most rate filing workflows were never built to preserve that second answer as a standing artifact. Feature engineering pipelines get rebuilt between model refreshes, vendor data contracts rarely specify documentation rights beyond delivery, and lineage tracking, when it exists, usually lives in a data engineering team's internal tooling rather than in a form an actuary can hand to a market conduct examiner. Closing that gap is a data infrastructure project as much as an actuarial one, and it is the single largest new compliance lift the tool introduces relative to what ASOP-based rate filing documentation already requires.

Where the Vendor Registry Meets the Exam Room

Exhibit D's reach into third-party data sits next to a second NAIC workstream moving in parallel: the Third-Party Data and Models (H) Working Group's proposed vendor registry. At its March 23, 2026 session, that working group sketched a registration regime, not a licensure regime, for vendors supplying AI models and datasets to insurers (Swept AI, March 2026). Registration requires a vendor to file information with regulators and update it on a defined cadence, and it creates regulatory visibility into a carrier's AI supply chain. It does not create a safe harbor. An insurer using a registered model remains fully answerable for that model's behavior in pricing, underwriting, claims, and fraud detection, and state departments have signaled they will treat the registry as a prerequisite for, not a substitute for, the carrier's own diligence file.

Once that registry exists, an Exhibit D request stops being a question the actuarial team answers alone. A pilot-state examiner asking for training data lineage can cross-reference the registered vendor's own filed disclosures against what the carrier represents in its Exhibit D response, and any gap between the two becomes its own finding. That is the practical consequence of the two workstreams converging: a carrier's AI vendor stack, once opaque to regulators by default, becomes independently examinable through the registry at the same time the evaluation tool is asking the carrier to document that stack from its own side. Realistic first-state implementation of the registry sits in late 2026 or early 2027, roughly the same window the evaluation tool is expected to move from pilot to adopted instrument (Swept AI, March 2026).

Why This Matters for Actuarial Teams

The industry's own comment letter, filed jointly by trade groups representing life, health, P&C, mutual, and reinsurance insurers on December 5, 2025, stated plainly that "the industry remains significantly concerned about the lack of detail and guidance around the proposed pilot" (Joint Trade Association Letter, December 2025). That objection was aimed at process, participation being effectively mandatory once a state sends a request, no fixed end date, and results potentially informing enforcement before the tool is finalized. The process objections may get resolved through the September and October 2026 revision cycle. The underlying documentation standard will not get easier; it will get formalized. The November 2026 adoption vote is the real deadline, not the pilot's September close. A tool adopted at the Fall National Meeting becomes the reference instrument examiners in bulletin-adopting states reach for by default, extending its practical reach well past the 12 pilot states within a single exam cycle. Actuarial teams that wait for their state to formally adopt the tool before building an Exhibit A model inventory, an Exhibit C high-risk classification methodology, and an Exhibit D lineage trail for their pricing and underwriting models will be building all three under exam pressure rather than on their own schedule. Building the lineage trail now, starting with the models most exposed to a material-financial-impact reading of "high-risk," is the work that turns a scored exam into a documentation exercise the team has already finished.