When Should an AI Decision Go Back to a Human?

AI systems can now tell you how sure they are. On November 11, 2026, Google released a model whose only job is to make bounded decisions and report, for every answer, a confidence score between 0 and 1. The same week, United Airlines decided that no AI agent, at any confidence level, may complete a ticket purchase for a passenger. Between those two events sits a question most teams running AI agents have not answered in writing: which decisions may the software settle on its own, and which must come back to a person?

A confidence threshold is the point on a model's 0-to-1 confidence scale below which an AI system stops acting on its own answer and routes the decision to a person or a fallback. It is not a safety guarantee and it is not a measure of truth. It is a routing rule, and like any routing rule it is only as good as the tiers, the review process, and the records behind it.

The short answer: let the machine settle what is cheap to get wrong, queue what is ambiguous, and keep anything that amounts to a signature with a person, at any score. Getting there takes four working parts: tiered actions, priced errors, a calibration run in shadow mode, and a review ledger. None of them needs a machine-learning team, and the hardest one is organizational: writing down, in advance, which mistakes the team will tolerate at what rate.

What a confidence score measures, and what it does not

A language model used for generation produces open-ended text. The same underlying technology used for judgment answers a bounded question: yes or no, which of these options, what grade on this rubric. Generation is hard to score. Judgment is easy to score, because the space of possible answers is closed. That difference is why judgment is where confidence numbers first became a product feature rather than a research topic.

The confidence score attached to such an answer is the model's own estimate of the probability that its answer is right. A well-calibrated model is one whose estimates match reality over many decisions: of all the answers it marks 0.9, about 90 percent turn out correct (calibration is measured over a test set, not promised per answer). Calibration is a statistical property. It says nothing about any single decision.

That distinction carries the whole subject. A score of 0.92 on one refund request does not mean the request was handled correctly. It means the model, if its calibration holds for inputs like this one, would be right on roughly 92 of 100 similar requests. Whether that is good enough depends entirely on what the other 8 cost you.

There is a second thing the number does not carry: responsibility. A confidence score is produced inside the model. Accountability for the action lives outside it, with whoever authorized the system to act. This is the core idea behind a sovereign AI avatar: an agent may propose and even execute, but what it is allowed to execute is judged outside the model, under rules its owner set. A threshold is one such rule. It is the owner's rule, not the model's.

Calibration, fortunately, is something a team can measure without any machine learning expertise. Take the last 500 decisions of the kind you want to automate, where you already know the right answer because a human handled them. Run the judgment layer over them. Sort the results into score bands: everything scored 0.90 to 0.95 in one bucket, 0.95 to 0.99 in the next, and so on. Then check each bucket against the known answers. If the 0.90 bucket is right about 90 percent of the time, the model is calibrated on your traffic. If the 0.90 bucket is right 72 percent of the time, you now know the dial reads high, and you know it before an incident teaches you (500 cases is a working minimum; smaller samples make the per-bucket percentages too noisy to act on).

The same exercise, repeated monthly on recent traffic, is drift detection. Calibration decays for mundane reasons: the product changed, a new customer segment arrived, the busy season altered what a typical request looks like, the vendor silently updated the model. None of these announce themselves. The only notice you get is the gap between the bucket percentages this month and last month, which is why the measurement has to be scheduled rather than occasioned. A one-off calibration check at rollout is a photograph; what the threshold needs is a pulse.

One practical note on where the number comes from. A dedicated decision model produces its score from the probabilities it computes internally over a closed set of answers. A general chat model asked to "also output your confidence" produces a plausible-sounding number with no such grounding, and published evaluations have repeatedly found such self-reported numbers poorly calibrated. The difference matters when you choose tooling: a score you will route real actions on should come from a system built to produce scores, or be validated as if it were not.

When the dial reads high: reported confidence vs. measured accuracy (example scenario)

Google now sells judgment separately from generation

Google's announcement is worth reading closely because it turns an architectural opinion into a priced product. The Gemini Decisions API, released November 11, 2026 and powered by a dedicated model called Gemini Decision 1, does not generate prose at all. It evaluates up to 1,000 bounded questions against one input in a single pass and returns probabilities.

The shape of the product, from Google's own launch post (November 11, 2026; Google is the vendor here, so performance figures are the vendor's own):

Two lines in that table matter more than the rest. The price first: at $0.10 per million input tokens, a policy check over a 2,000-token email costs $0.0002 (assuming 2,000 input tokens per check and no retries). Five thousand such checks cost a dollar. Judgment, as a product, is now cheaper than almost anything else in the AI stack, including the generation it supervises.

The second line is the design assumption hiding in plain sight. Google's own documentation says the confidence score exists "so your code can route ambiguous cases to a fallback or human reviewer." The vendor that sells the judgment is telling you, in the product manual, that some judgments should not be accepted automatically. Google reports using the same API internally to route everyday queries to lighter models, cutting latency by up to 40 percent on those queries, and to drive rapid design iteration in its Stitch canvas tool (both Google's own figures and examples, November 11, 2026, from the launch post). How that routing layer works in general, and why a judgment model is a natural fit for it, is the subject of multi-model routing.

The zero-output-token pricing has a business reading worth pausing on. Billing nothing for the answer means Google expects the money in volume: judgment is being positioned as plumbing, called thousands of times per workflow, the way a database index is consulted rather than the way a report is commissioned. For buyers this cuts two ways. Cheap judgment makes threshold routing affordable at any scale, which is good. Cheap judgment also removes the cost friction that used to make teams think before wiring a model's opinion into yet another automatic step. When every check costs a fiftieth of a cent, the temptation is to let the checks replace the checker everywhere, and the discipline of deciding which tiers still need a person has to come from policy, because it will no longer come from the invoice.

None of this is an endorsement of one vendor's scores. Any judgment layer, bought or built, gives you the same three ingredients: a bounded question, an answer, and a number describing how much the answerer trusts itself. Tiers, thresholds, and ledgers need nothing more than those three.

The build-or-buy question therefore reduces to ordinary engineering trade-offs rather than anything exotic. Buying a dedicated decision API gets you trained calibration and millisecond latency, at the price of a new external dependency in the most sensitive position in your stack: the component that decides what your agents may do. Building means running an open-weight model with a constrained output head, or a small fine-tuned classifier, on hardware you control. The score quality starts lower and the maintenance is yours, and in exchange the judgment layer cannot be repriced, deprecated, or silently updated by someone else's roadmap. Cost rarely decides it. At $0.10 per million input tokens (Google's November 11, 2026 pricing), even a team running 10,000 policy checks a day on 2,000-token inputs spends about $2 a day, or roughly $730 a year (10,000 × 2,000 tokens × $0.10/1M × 365, ignoring retries). Self-hosting a small judgment model costs more than that in electricity alone before counting the engineer who maintains it. What actually decides the question is the sensitivity of the decisions and the cost of the dependency: a team routing public content tags can buy judgment happily; a team routing payment approvals may conclude the judging component belongs with the party that answers for the payments. There is a middle path many teams land on: buy the judgment for low-stakes tiers, and keep the high-stakes tier on human review plus a self-controlled checklist, so that no external API sits between an expensive action and the person who owns it. The tier structure in the method below makes that split natural rather than awkward.

A check costs less than the task it checks (log scale)

Some decisions stay with a person at any confidence level

The same day Google priced judgment at $0.10 per million tokens, United Airlines drew a line that no confidence score can cross. The Wall Street Journal reported, and The Verge confirmed with United on November 11, 2026, that the airline will not let third-party AI agents complete ticket purchases. Agents may still search and display flights. They may not transact.

United's reasons are more instructive than the ban itself. Privacy is one. Unintended purchases are another: an agent buying a ticket its owner did not mean to buy. The third is the sharpest: federal rules require passengers to confirm certain declarations themselves, such as attesting that their luggage contains no explosives. An agent cannot make that attestation for you, because the attestation is the legal act of a person accepting consequences. CEO Scott Kirby added a commercial motive, saying "To be honest, I don't want to create another middleman." United also said it may reconsider as traveler protections mature, and the Verge piece notes eBay and several major banks have placed similar restrictions on agents.

Strip away the aviation specifics and a general category appears: decisions that are attestations. Signing a contract. Declaring information true under penalty. Accepting liability. These are not hard questions a better model will eventually answer. They are acts whose meaning is that a particular person performed them. A confidence score is irrelevant to them in the way a very accurate forged signature is still a forgery.

So before any threshold arithmetic, a team needs a short list of decisions that are simply off the automation table. Most teams discover theirs is shorter than they feared, which makes the remaining territory easier to automate with a clear conscience.

The counterparty angle deserves a moment too, because United's ban binds other people's agents, not its own software. If your team's avatar books travel, files listings, or transacts on third-party platforms, part of your threshold policy is now set by counterparties: eBay has moved against agent browsing, several major banks restrict agent access to accounts, and more sectors are visibly deciding (The Verge, November 11, 2026). A workflow that automates perfectly today can hit a counterparty wall tomorrow, and United's own statement that it may reconsider "as third-party AI agents advance" cuts both ways: the walls move. The practical consequence for planning: treat "the counterparty refuses agents" as a tier of its own, with a human fallback path that is tested, not theoretical, so a policy change at an airline or a bank degrades your workflow instead of halting it.

What a well-set threshold buys you

The honest case for thresholds is volume. A team letting agents handle autonomous workflows such as monitoring, triage, or first-draft contract review faces hundreds of small decisions a day. Humans reviewing all of them stop reading by Thursday. Humans reviewing none of them find out about the bad ones from a customer. A threshold is the instrument that makes the middle position concrete: the machine settles the calls it is demonstrably good at, and the ambiguous ones arrive in a queue sized for actual human attention.

It also converts a vague anxiety into a number someone can own. "Are we comfortable letting the agent approve refunds?" is not decidable in a meeting. "Below 0.90 it queues for Dana, and here is last month's error rate above 0.90" is decidable, auditable, and adjustable. The threshold becomes a policy object: it can be reviewed, versioned, and tightened after an incident, the way firewall rules are.

There is a quieter benefit. A routed decision leaves a trail. When a person approves the agent's low-confidence call, the record shows what the model estimated, what the person decided, and when. Over months this trail becomes training material for the team, evidence for auditors, and the raw data for the only question that matters at renewal time: is the automation actually right at the rates its scores imply?

A threshold, finally, is cheap insurance against your own optimism. Teams consistently overestimate how well deployment conditions match the pilot. The threshold does not prevent that mistake. It bounds the cost of discovering it.

It helps to picture the difference on an ordinary Tuesday. Without a threshold policy, a support lead either reads every agent-drafted refund before it goes out, which means the agent saved her composition time and none of her judgment time, or she spot-checks when she remembers to, which means the agent's worst calls ship on her busiest days, precisely when spot-checking lapses. With the policy, her morning queue holds the eleven cases the model scored low, each with the score and the reason on top. She clears them with coffee, disagrees with two, and the disagreements go into the month's calibration data. The agent handled the other hundred and forty. Nobody checked them, and that was a decision someone made deliberately, in writing, with numbers, rather than a gap nobody noticed.

What a threshold cannot do

The failure modes deserve names, because named risks get managed and unnamed ones get discovered in incident reviews.

Calibration risk is the gap between the score and reality. A model calibrated on last quarter's tickets can drift when the product, the customers, or the input language changes. A 0.95 after drift may be worth what 0.80 was before. The number on the dial moved; the needle underneath did not. Calibration has to be re-measured on your own recent traffic, on a schedule, not assumed from a vendor benchmark. Google's calibration claims for Decision 1, like any vendor's, were measured on Google's benchmarks (vendor figures, November 11, 2026), not on your refund queue.

The accountability gap is the harder one. Routing a decision to a human puts a person in the loop. It does not by itself make that person responsible, informed, or awake. If the review queue delivers forty items an hour to someone also doing a day job, review decays into rubber-stamping within weeks, and the organization has bought the appearance of oversight at the price of real oversight. The fix is unglamorous: cap queue volume, measure disagreement rates (a reviewer who never disagrees with the model is not reviewing), and make the approval an act someone signs with their name.

Distribution shift is the third named risk: confidence scores are least trustworthy exactly where decisions are most unusual, because unusual inputs are the ones the model saw least in training. The cases most likely to deserve human eyes are the ones where the score itself is least reliable as a summoning mechanism. Low-confidence routing catches much of this, and a hard ceiling (certain categories always route, regardless of score) catches more.

Aggregate risk is the fourth, and the easiest to miss because every individual decision looks fine. A thousand auto-approved $30 refunds at a 1 percent error rate is ten wrong refunds, about $300, probably acceptable. The same machinery auto-tagging a thousand customer records with a 1 percent error rate may quietly corrupt a dataset that a quarter's reporting is built on, and no single decision in the batch was large enough to route to anyone. Thresholds operate per decision; some harms only exist in the sum. The guard is a periodic sample audit of the auto-approved band, exactly the traffic the threshold exempted from review, because that band is where volume lives and where drift compounds silently.

Be clear about scope as well. A threshold addresses mistake risk on routine, bounded decisions. It does not address attestation decisions, which United's case shows must stay personal; it does not address vendor risk, since the judgment layer itself can be discontinued or repriced like any API; and it is not a kill switch. Stopping an agent that is mid-action is a different control, with its own requirements, closer to a real AI off switch than to a routing rule.

How to set a threshold in practice

The method assumes nothing about vendor choice. It needs a judgment layer that returns scores, a place to send queued decisions, and someone willing to own the numbers. Teams buying rather than building can fold these questions into a procurement standard before signing.

A note on who does this work. In a large company the exercise lands with a platform or risk team; in a ten-person company it is one afternoon for whoever runs operations, and it is still worth the afternoon. The tier list is a conversation, the error pricing is arithmetic on a whiteboard, and shadow mode is a spreadsheet column. None of it requires machine learning staff. What it requires is the willingness to write down, before the first incident, which mistakes the team has decided to tolerate at what rate, so that the eventual incident is a data point inside a policy instead of a referendum on the whole idea of automation.

Tier actions by blast radius, not by frequency

List what the agent can actually do: send, publish, purchase, delete, reply, file. For each action, ask what the worst plausible wrong execution costs, in money, in trust, in legal exposure. Three tiers usually suffice. Reversible and cheap (an internal draft, a tag): the agent acts, no score consulted. Reversible but visible, or costing real money (a customer reply, a small refund): the agent acts above the threshold, queues below it. Irreversible, expensive, or an attestation (a payment past a limit, a contract signature, a deletion without backup): a person acts, always, and the agent's role ends at preparing the decision. The attestation tier is United's lesson applied at desk scale.

Where 600 weekly requests land, by confidence score (example scenario)

Price both kinds of error before picking a number

Every threshold trades two costs. Set it high and humans review mostly correct calls: you pay salary for reassurance. Set it low and errors ship: you pay in refunds reprocessed, apologies, cleanup hours. Write both prices down, even roughly (a wrong $40 refund costs the $40 plus about 20 minutes of makegood; a needless review costs about 90 seconds of a reviewer's attention). The threshold that balances those two numbers is a budgeting decision a line manager can own and defend, which is exactly what a policy needs to survive its first incident review.

The threshold is a cost minimum (example scenario)

Start conservative, in shadow, then earn autonomy downward

Run the first two to four weeks in shadow mode: the judgment layer scores everything, humans still decide everything, and nobody's work changes. Shadow traffic gives you the two curves every later argument rests on, the score distribution and the human-model disagreement rate by score band. Then switch on automation only for the top band (0.95 and above is a common opening stance, but your disagreement curve, not convention, should pick it) and widen the band only as measured error rates earn it. Autonomy is granted downward from caution, never patched upward from an incident.

Six weeks live, one drift caught (example scenario)

Log every routed decision as if someone will read it

Each queued decision should record: the input (or a durable pointer to it), the question asked, the model's answer and score, the threshold version in force, who reviewed, what they decided, and when. Thin records make every later question unanswerable, from "has drift started?" to "who approved this?" The ledger matters more than the threshold, because the threshold will change and the ledger is how you will know it should. Where that ledger lives, and whether it survives a vendor change, is a question of agent memory ownership; a review trail locked inside a vendor's console is a trail you may one day lose.

Version the threshold and give it one owner

A threshold that nobody owns gets changed in a hurry and remembered by nobody. Treat it like a firewall rule: it has a current value, a change history with reasons, and a single named owner with the authority to move it. Changes outside an incident go through the owner with a one-line rationale in the log ("drift detected in auto band, 0.95 → 0.97, recalibration scheduled"). Changes during an incident are allowed in one direction only, toward more human review; loosening waits for the postmortem. Two further habits pay for themselves. Review the threshold on a calendar, quarterly at minimum, against the latest calibration run, so the value reflects current traffic rather than the optimism of launch week. And when the owner changes jobs, hand the threshold over explicitly, with the ledger, the way on-call rotations hand over pager duty; an orphaned threshold is how organizations end up unable to say who decided that the machine may spend money.

Worked example

Consider a 40-person online retailer whose support avatar handles refund requests (600 requests a week, averaging $38 each). The team tiers the action "approve refund" in October: automatic above 0.95 for amounts under $50; queued to the support lead between 0.70 and 0.95, or for any amount from $50 to $200; always human above $200 or for any order flagged for fraud, whatever the score.

The error pricing behind those lines was a twenty-minute exercise. A wrongly approved refund costs the amount itself plus about twenty minutes of a support agent's cleanup when the customer or finance notices; the team wrote that down as roughly $55 all-in for a typical case. A needlessly queued request costs the lead about ninety seconds. At those two prices, sending a borderline $35 request to a human pays for itself if even one in forty such reviews catches a real error, which the shadow data said it comfortably would. Shadow mode also produced one unbudgeted finding: the model scored gift-card refunds confidently and got them wrong at four times the overall rate, so gift cards were moved to the always-human tier by category, a fix no threshold value could have delivered.

Four weeks of shadow mode, then six weeks live, produce the numbers the team actually governs with:

In January the product line changes and the error rate in the auto band creeps to 3 percent. Because the ledger records scores alongside outcomes, the drift shows up in the weekly report, the threshold moves to 0.97 within a day, and recalibration gets scheduled. Without the ledger, the first sign of drift would have been a refund dispute. The team keeps the tier list and the current threshold in the operations runbook, next to the on-call rota, so the next hire learns where the lines are on day one.

What changed in the retailer's week (example scenario)

A short evaluation before you turn anything on

The questions compress into a table a team can walk in one meeting. Hesitation on any row means the system is not ready for autonomy on that action.

How KarmaBox applies confidence thresholds and human review

Our own product, KarmaBox, is a platform and network for running your own AI avatars, and the working parts above, tiers, approvals, and the ledger, all have direct counterparts in it.

The tiering step is native: each AI avatar carries owner-set policies per action class, so sending, publishing, spending, and deleting each have their own rule rather than one global switch. The always-human tier takes the form of approval cards: an action above the owner's line does not execute silently; it arrives as a card the owner taps to approve or reject, which keeps the attestation-class decision with the person who owns its consequences. The judgment layer is configurable in the same spirit as the routing uses Google describes: avatars can run decision checks before acting, and teams choose where that judgment and the work itself run, keeping sensitive review flows on machines they deploy themselves while public-facing tasks run on cloud avatars. The ledger step is covered by design: what an avatar did, what was approved, and by whom stays in the owner's records rather than a vendor console, and the configuration steps for policies and approvals are documented in the product docs. Teams rolling this out across departments usually start from the business deployment path, where per-team action policies matter most.

What KarmaBox does not change is the method's own homework: tier lists, error prices, and calibration runs are judgment calls only the owning team can make. The platform enforces the lines; the team still has to draw them.

FAQ

About the author

Karma Box Team. We publish practical guides on AI sovereignty, workspace control, and vendor evaluation. This guide separates reported facts (sourced and dated in the sentence where they appear) from our judgment, quotes vendors' own framing where it cuts against our argument, and keeps KarmaBox-related content in its own clearly marked section. It does not certify any vendor's calibration claims and is not legal or compliance advice. News facts checked on November 12, 2026.