In February 2024, a tribunal in British Columbia ordered Air Canada to pay a customer C$812.02. The airline's website chatbot had told him he could book a full-fare flight to his grandmother's funeral and claim the bereavement discount afterward. The airline's own policy page said the opposite. The money was small. The lesson is most of AI oversight in one case: the chatbot looked low stakes, and it wasn't.
Here's the claim. Triaging AI oversight by stakes is right, and most teams do it wrong, because they judge stakes by what an output looks like. Stakes live in what an output can reach.
In issue 002 I argued you can't verify everything, so you sort your agents' output by what it would cost to be wrong and spend your scrutiny there. I still hold that. What that advice skips is the hard part: who decides the cost, when, and by looking at what.
Stakes are not in the output
Put two outputs in front of a reviewer. One is a refund approval: a dollar figure and an approve button. The other is a friendly paragraph answering a customer's question. The refund gets the scrutiny. It looks like money.
But a paragraph that tells a customer they'll get a refund is a refund approval. It's just dressed as conversation. The tribunal in the Air Canada case didn't care that the promise came from a chatbot instead of a static page. The airline was responsible for what its website said, either way.
So stop asking what the output is. Ask what it can reach. In practice that's three things: money, a commitment made in your name, and another system that acts on it next. If an output can touch any of those, it isn't low stakes, no matter how ordinary it looks.
Triage before the output exists
The common way to triage is by eye. A reviewer reads each output and decides how hard to look. That puts the stakes call at the worst possible moment: one item at a time, mid-queue, with a hundred more waiting.
Make the call in advance instead, per action type, not per output. For every kind of action your agent can take, write down two things. The worst plausible outcome. And the undo window: how long before the action can't be taken back.
The undo window does most of the work. A draft sitting in an outbox can be pulled back any time until it sends. Once it sends, the window is zero. A record your agent writes at noon may be easy to fix until a nightly sync copies it into three other systems, and after that, fixing it is a project. A chat answer has a window of roughly zero, because the customer acts the moment they read it.
Then the tiers follow.
If the action reaches none of the three, sample it. A random slice each week, read properly, beats skimming everything.
If it reaches one but the undo window is long, review before the window closes, not before the action. Batch it. You keep the speed.
If the undo window is near zero, gate it. A human approves it before it ships, or a hard constraint makes the bad version impossible: an agent that can only quote policy text word for word can't invent a refund rule.
Tier by the shortest undo window an action can reach, not the typical one. The typical case is not where you get sued.
A tier you can defend
Regulators are moving the same way. In April, US bank regulators replaced their 15-year-old model risk guidance, SR 11-7, with SR 26-2, which leans on a risk-based approach tailored to each bank's own model risk profile. Expect the question that follows any risk-based rule: why is this thing in that tier?
A tier written down per action type, with its worst case and undo window next to it, answers that question. A tier decided by a reviewer's glance doesn't. That's not a tier. It's a mood.
Twenty-five years of production automation taught me the same rule from the other side: the controls that held were attached to what a step could change, not to what its output looked like.
Spend your scrutiny where an output can reach, not where it looks important. The paragraph that promises a refund deserves more attention than the dashboard that displays one.