How to set confidence thresholds and escalation in AI agents

Turn a decision model's confidence scores into measured thresholds that decide when an agent acts, falls back, or escalates.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

This skill decides which of an agent's decisions it may act on alone, which drop to a fallback, and which go to a person or a stronger model. In a Jev-style agent, each decision call returns a typed answer, a probability for every option and a confidence score. Those numbers are the raw material. The skill is turning them into a policy you can defend. For background on the overall split between writing, deciding and acting, see the Jev Engineering method page.

The starting premise is that confidence is not correctness. Coverage of the method notes that a typed answer cannot fall outside the requested type but can still be wrong, which is exactly why confidence scores matter. The practical consequence comes from one implementation guide, which tells builders to measure observed accuracy by probability band instead of trusting that a stated confidence already matches reality.

The published evidence shows why you have to measure your own case. One report measured a ten-bin expected calibration error of 0.0313 on a 1,200-item MMLU sample, which suggests well-behaved confidence on general questions. Another test, on 900 synthetic support tickets whose deciding policy was absent from the text, found an expected calibration error of 0.107 against a 0.024 noise floor, roughly 4.4 times worse. The two sources do not contradict each other so much as mark a boundary: confidence tracked accuracy when the needed information was in the input, and drifted when the rule that decided the answer lived somewhere the model could not see. Your production decisions may sit on either side of that line, and only your own data tells you which.

The output of the skill, for each decision point, is a short set of artifacts: a band-by-band accuracy table built from real outcomes, a threshold per branch, a list of conditions that trigger a fallback, a named escalation route, and a log format that lets you replay any decision later. The sequencing principle is conservative. One rollout guide advises automating the safest branch first while uncertain cases stay with a human or a stronger model, so the agent earns autonomy one branch at a time rather than all at once.

How It Works

A threshold policy has four inputs: a decision with a fixed answer space, a set of recorded decisions with known correct answers, a cost of error for each branch, and an escalation target that can absorb whatever the agent does not handle. The mechanism that connects them is the probability band.

To build bands, take every recorded decision, bucket it by the confidence the model reported, and compute how often the answer in each bucket was actually correct. The implementation guide frames this as measuring observed accuracy by probability band, and the rollout guide describes the same step as plotting accuracy against confidence and setting thresholds from that data. If the bands line up, a stated confidence means roughly what it says. If they do not, the table tells you what each confidence level is really worth, and you set the threshold from the observed column, not the stated one.

The recorded decisions should come from running the model beside the existing workflow before it controls anything. One practitioner describes logging the Jev choice next to the current fixed behavior and then calibrating the threshold with rework, human escalation and task success, which keeps live behavior constant while the evidence accumulates. How to run that comparison and measure the full loop is covered in Benchmarking and Observing Agent Loops; here the point is that thresholds are only as good as the outcome labels behind them.

Every call then resolves into one of four paths. A request that fails or returns output that does not parse never reaches the threshold check. The guide lists request failure, malformed output and insufficient savings as the fallback triggers, so a call that errors, returns garbage, or costs more than the path it was meant to replace drops to the existing behavior. A well-formed answer at or above the threshold is automated, and code still executes and gates the action. A well-formed answer below the threshold escalates, following the advice that uncertain cases remain with a human or a stronger model.

flowchart TD
  A[Decision call] --> B{Request succeeded}
  B -- no --> F[Fallback path]
  B -- yes --> C{Output well formed}
  C -- no --> F
  C -- yes --> D{Above branch threshold}
  D -- yes --> E[Automate action]
  D -- no --> G[Escalate to human or stronger model]
  E --> H[Log decision and outcome]
  F --> H
  G --> H

Thresholds are per branch, not per model. A choice between two low-stakes options can automate at a lower band than an approval that triggers a refund, because the cost of a wrong answer differs. Some branches may never automate at all, and that is a valid result of calibration.

Finally, every decision is logged with enough context to replay it. The guide specifies model version, question version, threshold, action and overrides. Without the versions you cannot tell whether an accuracy shift came from the model, from a reworded question or from a threshold change, and without the overrides you lose the most useful signal a human reviewer produces.

Step-by-Step Guide

Step 1: Choose one bounded, low-risk decision

Pick a single decision point whose possible answers are clear and limited, such as picking between two workflows or flagging a ticket for review. The rollout guide recommends starting with one bounded, low-risk decision with clear possible answers. Avoid decisions that trigger irreversible side effects on the first pass. Write down every option in the answer space and what a correct answer looks like for each, so outcomes can be labeled later without debate.

Pro tip: If two reviewers would disagree on the correct label for more than a handful of cases, fix the rubric before calibrating, or the bands will measure rubric noise.

Step 2: Price each branch's errors

For every option the decision can take, write down what happens when the agent picks it wrongly: a wasted retry, a customer-visible mistake, a compliance issue. This cost determines how strict the threshold must be on that branch. A branch whose error is cheap and reversible can automate at a lower observed accuracy than one whose error reaches a customer. State the required accuracy per branch as a recommendation you will revisit, for example 90% for a reversible routing choice and 99% for anything that sends a message externally.

Pro tip: Record the required accuracy before you look at any model output, so the threshold is not reverse-engineered from whatever the data happens to show.

Step 3: Collect outcomes without changing behavior

Run the decision model beside the current workflow and record its answer, its confidence, what the existing system did, and what eventually turned out to be right. The implementation guide advises that you start in shadow mode without changing real behavior. One practitioner calibrates the threshold using rework, human escalation and task success as outcome signals. Include ambiguous and adversarial inputs deliberately, because those cases populate the low bands you most need to understand.

Pro tip: Aim for enough labeled cases that every band you plan to use holds a meaningful count, for example at least 30-50 per band, before trusting its accuracy figure.

Step 4: Measure accuracy by probability band

Bucket the recorded decisions by reported confidence, for example into ten equal-width bands, and compute the share of correct answers in each. This follows the guide's instruction to measure observed accuracy by probability band. Compare observed accuracy with the band's stated confidence. Large gaps tell you the confidence scale is miscalibrated for this decision, which one test found happens when the deciding policy is absent from the input.

Build the table separately for each branch if their error costs differ.

Step 5: Set thresholds and automate the safest branch

For each branch, set the threshold at the lowest band whose observed accuracy meets the required accuracy you wrote down in the pricing step. Turn on automation only for the branch where errors are cheapest and observed accuracy is highest. The rollout guide recommends that the safest branch be automated first while uncertain cases stay with a human or a stronger model. Leave the other branches in shadow or escalation until their data supports a threshold.

Pro tip: Start with a threshold one band stricter than the data strictly requires, then loosen it after a week of live outcomes confirms the band's accuracy holds.

Step 6: Wire the fallback triggers

Add code paths for the three fallback conditions the guide names: request failure, malformed output and insufficient savings. Each should drop the decision to the existing behavior, not to an escalation queue that may not be staffed for volume. Test each trigger explicitly by forcing timeouts, feeding responses that fail parsing, and simulating a case where the decision call costs more than the path it replaces. Confirm that each path is logged distinctly from low-confidence escalations.

Pro tip: Track fallback rate as its own metric; a rising rate often signals an upstream outage or a schema change before accuracy numbers move.

Step 7: Define the escalation route and log format

Name who or what receives below-threshold cases: a human review queue, a stronger generative model, or both in sequence. Specify what context goes with the escalation so the receiver does not have to reconstruct state. Log every decision with the fields the guide lists, model version, question version, threshold, action and overrides. Human overrides of automated decisions are the fastest way to spot a band whose accuracy has slipped.

Pro tip: Feed human decisions on escalated cases back into the labeled set; they are exactly the hard examples the low bands need.

Step 8: Recalibrate when inputs change

Treat thresholds as tied to a specific model version, question wording and input distribution. When any of those change, rerun the band table on the same fixed evaluation set before trusting the old threshold. The divergence between a 0.0313 calibration error on a general MMLU sample and a 0.107 error on tickets missing their deciding policy shows calibration depends on the input, so a new customer segment or a new policy can invalidate a threshold without any model change. Schedule a periodic recheck even when nothing obvious has changed.

Best Practices

• Set thresholds from observed accuracy, never from the model's stated confidence alone. The guide's advice to measure observed accuracy by probability band exists because stated and observed accuracy can diverge, and the gap is invisible until you measure it.
• Make sure the state contains the rule that decides the answer. When a deciding policy was absent from the ticket text, confidence stopped matching accuracy, so put the relevant policy into the input rather than hoping the model infers it.
• Calibrate per branch, not per model. Different options carry different costs of error, and a single global threshold either over-escalates cheap branches or under-protects expensive ones.
• Expand automation one branch at a time. Following the advice to automate the safest branch first keeps the blast radius of a bad threshold small and makes each change attributable.
• Keep fallback and escalation as separate paths. Failures and malformed output are infrastructure problems that should revert to existing behavior, while low confidence is a judgment problem that needs a reviewer; mixing them floods reviewers with outages.
• Version everything the threshold depends on. Logging model version, question version, threshold, action and overrides lets you replay a decision and tell which change moved accuracy.

Common Mistakes

• Treating a high confidence score as proof the answer is right.: Typed output only guarantees the answer's shape, and a decision can still be wrong. Check each band's observed accuracy before letting any confidence level trigger automation.
• Borrowing a threshold from a published benchmark instead of measuring your own decision.: Calibration results vary with the input, as the gap between one well-calibrated MMLU result and a poorly calibrated support-ticket result shows. Build the band table on your own labeled cases.
• Automating uncertain or high-risk branches before thresholds and fallbacks exist.: Run in shadow mode without changing real behavior until the band data supports a threshold, and keep uncertain cases with a human or stronger model in the meantime.
• Testing only the happy path and leaving fallbacks unexercised.: Force each of the named triggers, request failure, malformed output and insufficient savings, in a test environment. A fallback that has never fired is a fallback you do not actually have.
• Calibrating on clean examples only.: Include ambiguous and adversarial cases in the labeled set. Without them, the low-confidence bands are nearly empty and the threshold rests on the easy cases that never needed a model in the first place.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: Jev Engineering

Related Skills

• Benchmarking and Observing Agent Loops
• Formulating Atomic Decision Questions
• Separating Generation from Decision-Making
• Designing Routing and Ranking Policies
• Batching and Parallelizing Decisions
• Enforcing Deterministic Execution Boundaries
• Structuring Shared Agent State

Sources

• What is Jev Engineering?
• How to use Jev: first call in 5 minutes
• Jev: TypeSafe's Decision Model, Speed and Cost Explained
• Jev's Architecture Unmasked - archerhume
• Jev 是什麼？價格、限制與Agent 工作流的整合方向 - Wayneh
• Jev Agent engineering: separate decisions from LLM generation
• Akshay on X: https://t.co/haL8IGhx3h / X

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.