Joyjit Roy and Samaresh Kumar Singh posted arXiv preprint 2602.13213 on January 21, 2026, adding one agent to the standard agentic underwriting pipeline. A dedicated critic sits between the intake agent and the human reviewer and tries to refute the intake agent's reasoning before it advances.

Across 500 expert-validated commercial lines cases, that change cut the hallucination rate from 11.3% to 3.8% and raised decision accuracy from 92% to 96%. What it also produces is a document.

Key Takeaways

  • The critic cut hallucination from 11.3% to 3.8% and lifted accuracy from 92% to 96% across 500 expert-validated commercial lines cases.
  • On 10,000 annual commercial submissions, an 11.3% rate is roughly 1,100 decisions carrying materially unsupported AI reasoning before any human sees them.
  • The architecture's output is a structured artifact: the intake agent's position, the critic's challenge, and the resolution, which is what Article 9 of the EU AI Act asks for.
  • August 2, 2026 is when Annex III high-risk classification takes full effect for insurance underwriting AI in EU jurisdictions.
  • The critic is itself an unvalidated model, and its concurrence rate with the intake agent is the parameter that decides how much oversight the system actually delivers.

The Three Roles and What the Critic Can See

The architecture assigns three functions, and the middle one is the structural change.

An intake agent processes the submission, pulls structured risk data out of unstructured documents, and produces a preliminary recommendation. A critic agent receives that recommendation and applies adversarial pressure, looking for logical gaps, unsupported inferences, and factual inconsistencies in the reasoning. An arbiter, automated or human depending on configuration, resolves conflicts and escalates when they cannot be resolved algorithmically.

The critic's limits are worth stating precisely, because they define what the metric means. It does not re-examine the submission independently. It examines the intake agent's reasoning and attempts to refute it, which makes it blind to facts the intake agent never reported and able only to challenge inferences drawn from facts that were. A residual 3.8% is not a rounding error on the way to zero. It is three to four cases per hundred where the critic found nothing wrong and something was.

The 4-point accuracy gain is the companion figure. Decision accuracy here measures how often the system's recommendation matches what a qualified underwriter would reach on the same submission, so the move from 92% to 96% is roughly 400 additional correct recommendations per 10,000 submissions. Each of those is an avoided mispricing, an avoided adverse selection, or a coverage gap that would otherwise have surfaced at claim time.

One clarification the coverage of this paper tends to skip: 2602.13213 is a preprint. The 500-case result is a single published experiment, not a validated industry benchmark.

The Artifact Is Worth More Than the Percentage

The reason this matters on a compliance calendar rather than a research one is what the pipeline leaves behind.

The EU AI Act (Regulation 2024/1689) classifies insurance underwriting AI as high-risk under Annex III, covering AI used to assess and price life, health, and property casualty risk, with full effect from August 2, 2026. Article 9 requires a documented risk management system maintained as a continuous process, and it is specific about documentation: the technical record must support retrospective reconstruction of the reasoning behind each automated determination. A risk score and a timestamp do not satisfy it.

A three-agent pipeline generates that chain as a byproduct. The intake agent said X for reasons A, B, and C; the critic found that reason B rests on an unsupported inference about loss history; the arbiter escalated with both positions attached. Each element is timestamped in the operational log without anyone writing a compliance overlay.

A single-agent system produces a recommendation and no reasoned argument that survived challenge. Its human reviewer sees a conclusion without the reasoning that generated it, which is exactly the case that fails an audit: the carrier cannot evidence that the AI reasoning was sound on any individual file, because nothing in the system was built to record it.

Human sign-off does not close that gap. Annex III reaches systems that materially influence a pricing or coverage determination whether or not a human formally approves the output, and an underwriter clearing 200 AI-assisted submissions a day is not interrogating each reasoning chain from first principles. The efficiency case depends on not doing that. Duck Creek's AI Assurance layer, launched April 28, 2026, captures what agents did and when. The critic captures why a conclusion was reached and whether it held up. The two are complementary; platform logging alone yields timestamped decisions a regulator cannot evaluate for soundness.

The Critic Is a Model Nobody Has Validated Either

Adding an agent adds a model, and the second one is harder to characterize than the first.

The intake agent has a recognizable validation shape: training data, known failure modes, conditions under which output is reliable, performance on a representative test population. The critic's output is not a risk assessment but an assessment of an assessment, so characterizing its limits means specifying which categories of error it will miss, how its performance moves with submission type and complexity, and whether it introduces bias of its own. A critic calibrated on commercial property submissions is not obviously competent to challenge professional liability or surplus lines reasoning, where the relevant risk factors and document structures differ.

The governing parameter is the concurrence rate between intake and critic. At 98% concurrence the system escalates two cases per hundred, which may be right for standard renewals with low complexity variance. At 70% on the same population, either the critic is overreaching or the intake agent is producing unreliable recommendations at a rate that demands far more human attention than the carrier budgeted. Nothing in the architecture tells a carrier which reading is correct.

Direction of disagreement matters as much as frequency. A critic that consistently pushes recommendations toward decline or surcharge may be protecting underwriting quality, or may be carrying a systematic bias that Colorado's SB 21-169 fairness testing would surface as discriminatory. Either way it moves the loss ratio and the distribution relationship, and the two explanations are indistinguishable from the aggregate concurrence rate alone.

The escalation threshold has the same problem. When both agents reason in natural language rather than probability scores, quantifying disagreement against a threshold is itself a modeling choice, and the false-negative rate, the share of files carrying material intake errors that the critic never challenged, is the number that would tell a carrier whether the threshold is set right. Roy and Singh's 500 cases are the only published calibration reference available, on a population that is not any particular carrier's book.

Further Reading

Sources