AI Abstention and Uncertainty Handling for Decisions

Set per-task confidence thresholds from logged outcomes so software acts on reliable decisions and escalates uncertain ones to people.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

Abstention is the decision a system makes not to act on its own answer. When a decision model returns a typed answer plus a probability, the software around it has to choose, on every call, whether to execute the answer, hold it for a person, or fall back to a safe default. This skill is about writing that policy down and tuning it with evidence. For background on the training method behind these confidence values, see the RLCD method page.

TypeSafe frames the intended use plainly: its models are built to be used natively by machines, with software acting when confidence is high and escalating when it is not high enough. That framing puts the burden on the integrator. The model supplies a number. Your team decides what number is enough, for which decision type, and what happens below it.

The number only helps if it means something. TypeSafe says its training aims for calibration, so that when the model says 80%, it is right about 80% of the time across many cases. A calibrated score lets you predict how many wrong actions a given cutoff will let through. An uncalibrated score, however precise it looks, gives you a cutoff with unknown consequences. That is why this skill starts from logged outcomes rather than from the model's output alone.

There is no published universal threshold. The RLCD practitioner guide uses ten probability bins as an example but does not fix a sample size, a bucket count or an acceptable calibration error, and TypeSafe's public material does not specify an accept-versus-escalate cutoff either. Thresholds are local. They depend on what a wrong action costs, what a review costs, how many decisions arrive per day, and how well calibrated the model is on that specific task.

The outputs of this skill are concrete: a per-decision-type policy with an act threshold and an escalation route, a review queue that captures outcomes as a side effect of doing the review, and a monitoring loop that flags when a threshold has drifted out of safe territory. Done well, reviewers see a manageable share of genuinely hard cases and automated actions fail at a rate someone chose on purpose. Done badly, reviewers drown in easy cases, or bad actions slip through at a rate nobody measured.

How It Works

The act-or-escalate policy sits between the model and whatever executes the decision. Each call returns a structured decision and a probability. TypeSafe says each answer carries a probability distribution over possible outcomes and a confidence value describing how concentrated that distribution is. The policy reads that confidence, looks up the rule for the decision type, and routes the call to execute, review or fallback.

The rule is built from an empirical mapping. You log each prediction, its stated probability and the eventual true outcome, then group predictions into probability buckets and compare each bucket's mean stated confidence with the fraction that turned out correct, as the RLCD practitioner guide describes. That mapping from stated confidence to observed success rate is what tells you whether a candidate threshold is safe for automation. The same guide recommends measuring calibration separately for each question type, because calibration can vary by task, so you build one mapping per decision type rather than one for the whole system.

With the mapping in hand, a threshold becomes a trade-off you can price. For any candidate cutoff, take the logged decisions above it: their observed error rate times expected volume gives the number of wrong automated actions per period. The decisions below it give the review load. Raising the cutoff lowers wrong actions and raises review load, and lowering it does the opposite. A reasonable cutoff sits where one more review costs about as much as the errors it prevents. Because error costs differ sharply between, say, tagging a ticket and issuing a refund, thresholds differ by decision type.

Many teams use three bands instead of two. For example, a policy might act automatically above 0.9, send decisions between 0.6 and 0.9 to human review, and return a safe default or refuse below 0.6. The bottom band exists because some low-confidence cases are not worth a reviewer's time, or because the safe default (do nothing, keep the current state, ask the user) is cheap and harmless.

Calibration is a property of many predictions, not one. The practitioner guide describes it as a statement about repeated predictions at a confidence level, not a guarantee that each prediction at that level is correct. A threshold therefore controls the error rate of the automated stream, not the correctness of any single call. Decision types where one error is unacceptable may need review regardless of confidence.

The loop closes through the review queue. Every reviewed case yields a ground-truth label that feeds back into the mapping. You also need to sample decisions above the threshold for audit. Without that sample you only collect outcomes for low-confidence cases, and your estimate of the automated error rate silently goes stale as inputs change.

Step-by-Step Guide

Step 1: Inventory decision types and error costs

List every distinct decision the model makes, such as ticket category, refund approval or duplicate detection, and treat each as its own policy unit. For each, write down what a wrong automated action costs and what one human review costs, in money, time or risk. Note whether errors are reversible, since a reversible error can tolerate a lower threshold than one that ships to a customer. Record expected daily volume, because volume turns per-decision costs into queue sizes.

The output is a short table that the rest of the work fills in.

Pro tip: If two decision types share a label but have very different error costs, split them. Pooling them will force one threshold onto both and one of them will be wrong.

Step 2: Log decisions, confidence and outcomes

Run the model in shadow mode, or with every decision reviewed, and log the decision, its stated probability, the decision type, a timestamp and the eventual true outcome. The RLCD practitioner guide stresses recording outcomes before judging calibration, because a confidence value cannot be validated from the model output alone. Decide up front how the true outcome is determined, whether by reviewer verdict, a downstream event or a later correction. Keep logging until each decision type has enough outcomes to fill the confidence range you care about.

Pro tip: Log the full probability distribution if the model provides one, not just the top choice. It costs little and lets you revisit the policy later without rerunning traffic.

Step 3: Build a per-type confidence-to-accuracy mapping

For each decision type, group logged predictions into probability buckets and compare the mean stated confidence in each bucket with the share that was actually correct. This gives you the observed success rate at each confidence level for that task. Pay most attention to the high-confidence buckets, since those are the ones you will automate. If a bucket has too few examples to trust, widen it or keep collecting rather than guessing.

A type whose high buckets are much less accurate than stated is not ready for automation at those levels.

Pro tip: Read the mapping, not just a single summary number. An average calibration error can look fine while the one bucket you plan to automate is badly overconfident.

Step 4: Choose thresholds from costs

For each candidate cutoff, compute expected wrong actions above it and expected reviews below it, using observed accuracy from the mapping and the volumes from step one. Multiply each by its cost and pick the cutoff with the lowest total, then sanity-check that the resulting review load fits your team's capacity. Decide whether a bottom band should fall back to a safe default instead of review. Write the chosen thresholds, the data window they came from and the reasoning into the policy document so the next person can challenge them.

Step 5: Design escalation routes and the review queue

Define where each escalated decision goes: which queue, which role, what service time, and what the reviewer sees. Show reviewers the model's decision, its confidence and the inputs, and make recording the correct answer part of closing the item. Specify the fallback for when no reviewer acts in time, such as leaving the state unchanged. Route by decision type, since a refund dispute and a spam flag need different expertise.

The review queue is also your main source of new ground truth, so its outcomes must flow back into the log.

Pro tip: Consider hiding the model's confidence from reviewers on a sample of items. It reduces anchoring and gives you cleaner labels for measuring the model.

Step 6: Audit above threshold and monitor drift

Send a random sample of automated decisions to review so you keep measuring accuracy where the model acts alone. Recompute the per-type mapping on a schedule and whenever the model version, input sources or decision schema changes. Set an alert when observed accuracy in the automated band falls below the level your threshold assumed. When the alert fires, raise the threshold or pause automation for that type until you have fresh outcomes.

Keep a changelog of threshold moves and their reasons.

Pro tip: Size the audit sample so it produces enough outcomes per period to detect a meaningful drop, for example a few dozen audited items per decision type each week, and adjust once you see the noise level.

Best Practices

• Set thresholds per decision type, never globally. The RLCD practitioner guide notes calibration can vary by task, so one cutoff across tasks will be too loose for some and too strict for others.
• Derive every threshold from logged outcomes, not from the confidence scale itself. For example, a cutoff of 0.9 only means something once you have checked how often decisions at that level were actually right on your traffic.
• Price both sides of the trade-off explicitly. Writing down the cost of a wrong action and the cost of a review turns threshold setting from an argument into arithmetic that others can check.
• Keep a safe-default band for cheap, harmless fallbacks. Not every uncertain case deserves a reviewer, and a well-chosen default protects review capacity for decisions where human judgment changes the result.
• Always audit a sample of automated decisions. If you only collect outcomes from escalated cases, you lose sight of the error rate in the stream you actually automated.
• Require review for decision types where a single error is unacceptable. Calibration describes aggregate frequencies, so no threshold makes an individual high-stakes decision safe on its own.
• Treat model, input and schema changes as threshold resets. A mapping measured on the old version does not transfer automatically, so drop back to a conservative threshold until fresh outcomes arrive.

Common Mistakes

• Picking a round threshold such as 0.95 because it sounds safe, without checking outcomes.: Treat the model's number as a claim to verify. TypeSafe's stated goal is that 80% confidence means right about 80% of the time, but you still need your own logged outcomes to confirm that holds on your task before trusting any cutoff.
• Using one threshold for every decision the model makes.: Build a separate confidence-to-accuracy mapping and threshold for each decision type. Calibration can differ across tasks, and error costs almost always do.
• Measuring accuracy only on escalated cases.: Sample automated decisions for review as well. Otherwise your view of the automated error rate is based on old data and drift above the threshold goes unnoticed.
• Treating a decision above threshold as guaranteed correct.: A threshold bounds the error rate across many automated decisions, not the correctness of one. Design downstream actions to be reversible where possible, and require review for decision types where one mistake is unacceptable.
• Escalating without capturing the reviewer's answer.: Make recording the correct outcome part of closing a review item. The review queue is your main source of ground truth, and skipping this starves the mapping that keeps thresholds honest.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: Reinforcement Learning for Calibrated Decisions (RLCD)

Related Skills

• Calibrating Confidence to Outcomes
• Structuring Machine-to-Machine Decision Outputs
• Designing Schema-Constrained Decisions
• Designing Outcome-Based Reward Signals
• Estimating Decision Confidence
• Evaluating Probabilistic Calibration

Sources

• Home - TypeSafe AI
• RLCD explained: Reinforcement Learning for Calibrated Decisions
• RLCD - the training method behind Jev - Jev by TypeSafe AI

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.