The team that builds Claude Code merged more than twice as many pull requests between June and September while the share that had to be reverted fell by more than 40%, on a codebase Claude writes about 90% of, according to a post by Addy Osmani, Morgan Westlee Lunt and Qing Wang published on October 15 and circulated by Osmani on X on October 26. The post sets out seven layers a change passes through and one rule for choosing them: "did a human read the code?" is an imprecise proxy for "what could go wrong with this change, and which check is the right one to catch it?"
Anthropic's monorepo went from about 2,500 changes a day in May to more than 30,000 in September, the post says, and the share of internal pull requests receiving substantive review comments rose from 16% to 54% once Claude's review was switched on. The stated goal is to reduce human effort per change while keeping the revert rate no higher than it is now.
Actuarial code is arriving the same way, through agents writing reserving scripts and GLM pipelines faster than a reviewer reads them. The rules for that work predate the agents: ASOP No. 56, Modeling, effective October 1, 2020; the NAIC's December 2023 AI model bulletin, in force in 24 states and the District of Columbia; and the SR 26-2 bank guidance of April 17, 2026. Five of the seven layers have an equivalent in that rule set. Two do not.
Key Takeaways
- The seven layers are pre-PR verification, an AI reviewer on every pull request, a blocking security review, required CI, risk-scaled human approval, a merge queue with flags and staged release, and detection with rollback.
- ASOP 56 sections 3.6.1 to 3.6.3, model testing, output validation and review by another professional, cover the first two layers and the human one; the NAIC bulletin's 3.3(c) asks for repeatability, reproducibility, traceability and auditability, which is what CI enforces.
- The reviewer is tuned on incidents: past PRs that caused reverts or incidents form a daily test set, and after three rounds it asks the author to reconsider the approach; one PR ran more than fifty rounds overnight.
- About 59% of what the reviewer kept flagging predated its first pass, the equivalent of legacy spreadsheet logic nobody re-derived.
- The post's documentation is two files and a template: CLAUDE.md for conventions, REVIEW.md for what failure looks like, and a PR template covering testing, risk and how to revert, against ASOP 56 section 3.7 and the NAIC bulletin's inventory and documentation items.
- The actuarial incident is adverse development: $15.8 billion of it on US casualty in 2024, recognized years after the code that booked the reserves ran, with no flag to switch off.
Seven layers against three ASOP sections
The first layer runs before a pull request exists. Claude runs the tests, the build and the linters and shows the output, and a subagent reviews the diff in a fresh context. ASOP 56 section 3.6.1 describes model testing in the same terms: reconciling inputs to source systems, checking formulas and logic, sensitivity tests on key assumptions and reconciling results to prior runs, per the standard's text. The difference is who wrote the test. A reserving script usually has none, so the agent has nothing to run.
The second layer reviews every pull request with parallel agents that look for bugs and verifier agents that vote on each finding; only agreed findings are posted, tagged Important, Nit or Pre-existing, and each is fixed or resolved with a reason. ASOP 56 section 3.6.3 says the actuary "may consider obtaining a review by another qualified professional," and the NAIC bulletin's section 3.4 accepts "measuring performance against expert review" as validation. Both assume a person.
| Claude Code layer | Nearest actuarial rule | Fit |
|---|---|---|
| 1. Verification before the PR (tests, build, fresh-context subagent) | ASOP 56 3.6.1 model testing | Direct, where tests exist |
| 2. AI review on every PR (parallel reviewers, verifier vote, fix or resolve) | ASOP 56 3.6.3 review by another professional; NAIC bulletin 3.4 expert review | Partial: the rule assumes a person |
| 3. Security review that blocks the merge | NAIC bulletin 3.5 protection of non-public information | Exists at carriers, outside the actuarial function |
| 4. Required CI (lint, types, cross-platform tests, build) | NAIC bulletin 3.3(c) repeatability, reproducibility, traceability, auditability | Direct in principle; rare in practice |
| 5. Risk-scaled human approval (none, one or two people; some paths always human; an agent's approval never counts) | SR 26-2 materiality tiering; ASOP 56 3.6 mitigation scaled to purpose and complexity; ASOP 56 4.1 disclosures | Direct |
| 6. Merge queue, feature flags, employees first, latest before stable | None; a booked reserve or filed rate reaches everyone at once | Parallel run is the closest |
| 7. Fast detection and rollback; incident review asks which layer should have caught it | ASOP 56 3.6.2 outcome analysis and back-testing | The outcome arrives with the next valuation, or years later |
Sources: Osmani, Lunt and Wang, "How the Claude Code team does code review," October 15, 2026; ASOP No. 56; NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, December 4, 2023; Federal Reserve SR 26-2, April 17, 2026.
Layer four is where most actuarial repositories fall short of a rule they already have. The NAIC bulletin's section 3.3(c) lists "repeatability, robustness, regular tuning, reproducibility, traceability, model drift, and the auditability of these measurements" among the assessments a program should cover, and continuous integration is what makes repeatability true on every change rather than at audit time. The CAS's chainladder-python runs a pytest workflow and coverage reporting on each commit; most carrier reserving code has no equivalent, as the site's workflow guide set out in February.
Layer five is the one the profession already runs. A classifier sets the risk tier from the change, not the author: low risk merges on green checks with no human, medium needs one person, high needs two, and billing, the auto-updater and CI workflows always need a person. SR 26-2 replaced the uniform annual review with the same idea, effort scaled to a model's exposure and purpose, as the site's July translation described, and ASOP 56 section 3.6 scales mitigation to "intended purpose, complexity, governance, changes, and cost versus benefit."
Incidents are the eval, and the actuarial incident arrives late
The reviewer gets better from its own failures. Past pull requests that caused incidents or reverts, confirmed by the engineers who handled them, form a test set run against the reviewer daily, and noise is measured by what authors do with a comment: fixed, rejected, intentional or ignored. After three rounds the reviewer asks the author to reconsider the approach and later passes look only at the new diff; one CLI change went through more than fifty rounds overnight before that rule existed.
The actuarial version of that loop exists on paper. ASOP 56 section 3.6.2 names back-testing against historical results, hold-out data and comparison with alternative models as output validation, and Schedule P and the Actuarial Opinion Summary record how each year's booked reserves developed. The timing is the problem. US casualty lines produced $15.8 billion of adverse prior-year development in 2024 on accident years booked in 2021 and earlier, per the site's Schedule P review, and this quarter's earnings calls have carriers describing the 2023 and later years as too green to touch, as today's reserve piece reports. A reviewer trained on incidents gets a handful per decade, each confirmed three to five years after the code ran.
What can run daily is narrower: a reproducibility check that the same triangle and assumptions return the same reserve, the reconciliation to the prior quarter that 3.6.1 asks for, and the reviewer's Pre-existing tag. About 59% of what the reviewer kept flagging predated its first pass, the post says. In a reserving repository that is spreadsheet logic ported into Python without anyone re-deriving it, a defect an incident eval never sees because the incident is the reserve estimate itself.
Documentation that stays current, governance that scales
The post's last figure asks how anyone learns the code when machines do most of the reading. The team's answer is on-demand explainers, written by Claude for the slice of the codebase an engineer needs, in minutes and from the current code. ASOP 56 section 3.2 runs the other way: when expressing an opinion on a model's results, the actuary should understand its key operations, dependencies and sensitivities and its known weaknesses, and section 4.1 puts the intended purpose, material limitations and any reliance on others' models into the report. An explainer satisfies 3.2 only as far as the actuary can test it against the code, which returns the question to layer one.
The documentation the post describes is three artifacts, all of them read by the reviewer before a human sees the change. A short CLAUDE.md states the project's conventions. A REVIEW.md, written by code owners, states what failure looks like in their part of the codebase, caps the nits, lists the paths to skip and requires file-and-line evidence for every finding. A pull request template covers testing, risk and how to revert. None is a report; each is an input to the next check, and each is wrong the moment the code moves unless it is maintained with the code.
The actuarial documentation rules describe outputs instead. ASOP 56 section 3.7 asks for documentation "such that another actuary qualified in the same practice area could assess the reasonableness" of the work. The NAIC bulletin's section 3.3 asks for model inventories and "detailed documentation of the development and use" of each, section 2.2 for documentation of compliance with the program's own policies, and section 3.7 for a narrative of how the model is developed and validated, any of which a regulator may request on examination. None asks the REVIEW.md question, what failure looks like in advance: a tail factor outside its historical range, a triangle that no longer reconciles to Schedule P, a pattern that moved more than the prior year's. That is the document a reviewer, human or agent, would check a reserving change against.
On governance the fit is closer. ASOP 56 section 3.6.4 has the actuary use "reasonable governance and controls to mitigate model risk." The bulletin's section 1.3 vests responsibility with senior management, section 1.4 makes the program "tailored to and proportionate with" the insurer's reliance on AI, and section 3.1 asks for an approval process with "constraints and controls on automation." The post's risk classifier, with its hard-coded paths that always need a person, is sections 1.4 and 3.1 implemented as code, and its rule that an agent's approval never counts as a human's is the control the signing actuary's name already imposes on an opinion.
SR 26-2 excludes generative and agentic AI from its scope, with Vice Chair for Supervision Michelle Bowman saying on May 1 that it "applies narrowly to traditional models and basic AI applications," and the NAIC bulletin governs how an AI system is used, not how its code is built. The seven layers in the October 15 post sit in the gap between them, and two of the seven depend on a rollback a signed opinion does not have.