Should You Trust the AI?

The problem

A compliance software company had built an AI system that reviewed cases and issued a recommended verdict — approve, flag for review, or decline. It worked well enough in testing that leadership wanted to know if it was ready to make some decisions on its own, without a human checking every case first.

The question

How often does the AI’s verdict actually match what a human reviewer decides — and is “how often” even the right question to be asking?

The approach

Before writing a single query, the definition of “agreement” was pinned down against the product’s own specification, so the metric would measure what leadership actually meant, not a convenient approximation. Each case’s AI verdict was matched to its corresponding human decision, filtering out cases that hadn’t been decided yet and making sure a decision only counted if it came after the AI had actually run.

What I found

The AI and human reviewers agreed almost every time on approvals. On declines, agreement was barely better than a coin flip.

Averaged together, that would have looked like “pretty good, with room to improve.” Broken apart, it told a completely different story: the system could be trusted to approve, but not yet to decline.

What changed

That distinction reframed the automation roadmap. A wrong approval is far more costly than a wrong decline, so the rollout was sequenced around that asymmetry instead of a single blended accuracy number, and the specific false declines were flagged as the highest-leverage next piece of work.