TL;DR: Because insuring software that follows instructions is not the same underwriting problem as insuring software that makes decisions.
AI agents can interpret a situation, choose a response, and act. A harmful decision does not necessarily mean the software malfunctioned.
Let's take an example with a healthcare voice agent:
Why this deserves its own underwriting
Checking code and security is not enough. Insurers also need to assess what the agent can decide, what it can do without approval, and how much harm one decision could cause.
That's why, at Klaimee, we've built an underwriting system and risk model specifically for AI agents. We look beyond code and cybersecurity to understand what an agent can decide, what it can do without approval, and how much harm a wrong decision could cause. We then test the controls and guardrails in place: does the agent stay within its permissions, respect approval requirements, and escalate when it should? Those findings inform the scope, limits, and terms of coverage. A distinct risk needs underwriting built around its own parameters.
That same approach is the foundation of Klaimee's E&O liability policy, designed for agentic systems rather than legacy software. The starting point isn't just whether the technology might malfunction. It's the work the agent performs, the judgment it exercises, and the liability its decisions and actions can create, even when the software is working as designed. Think professional liability for AI agents: coverage built around the responsibilities they take on, not just the code they run.
Why your Tech E&O may not be enough
Coverage can be uncertain where the agent's activities fall outside the policy's insured services or an exclusion applies. A policy covering software errors should not be assumed to cover every autonomous action. Some policies expressly address AI; the wording matters.
Two questions for your broker
- Does our policy cover a harmful autonomous decision even when there is no software bug or cyber incident?
- Do its covered activities, exclusions, and conditions match what our agent actually does?
Originally published by Ines Boutemadja on LinkedIn.
Related reading: What is AI agent insurance? and What is AI agent risk?.