Steps for evaluating probabilistic calibration of AI models

Check whether a model's stated probabilities match real outcome rates with probability bins, reliability plots and expected calibration error.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

Evaluating probabilistic calibration of AI models means checking whether the numbers a model attaches to its decisions behave like frequencies. Accuracy tells you how often the model is right overall. Calibration tells you whether each confidence level means what it says. Public descriptions of RLCD define the property at the population level: if many predictions receive a probability of 0.2, roughly 20% of them should be correct, as this three-day review of Jev explains. For background on the training method that targets this property, see the RLCD method page.

You cannot take calibration on trust from a vendor. An independent review found no paper, no reliability curve, no expected-calibration-error number and no ablation separating RLCD from the architecture, and the same guide notes that TypeSafe's own documentation narrows the claim to groups of predictions. The same review of Jev also reports that no calibration curves have been published. Whatever model you run, the calibration evidence that should drive your automation decisions is the evidence you compute on your own logged traffic.

The skill rests on three instruments described in the RLCD practitioner guide: probability binning, the reliability plot and expected calibration error (ECE). The input is a log of decisions, each with its stated probability, its task type and the outcome observed later. The output is a binned table per task type, a plot that shows where the model is overconfident or underconfident, and a single summary score you can track over time.

Keep the scope honest. Calibration is a statement about many predictions, and Anthony Maio's analysis stresses that it does not tell you whether one particular answer is right. Your evaluation therefore judges bins and task types, never single answers. You will know it went wrong if your conclusions rest on a handful of predictions in a sparse bin, if different kinds of decisions were pooled into one score, or if you reported accuracy and called it calibration.

How It Works

Start from a log where each row holds a decision, the probability the model stated for it, the task type, and the outcome observed later. The RLCD practitioner guide describes the core procedure: sort predictions into probability buckets, for example ten bins, and compare each bucket's mean stated probability with the fraction of its predictions that were correct.

Probability binning. Equal-width bins split the probability range into even slices, which makes plots easy to read but can leave some bins nearly empty when a model clusters its confidence near the top. Equal-mass bins put the same number of predictions in each bin, which steadies the estimates but makes bin edges uneven. Pick one scheme per report and state it, because ECE values from different schemes are not directly comparable.

Reliability plot. Plot each bin's mean confidence against its observed accuracy. The guide describes calibrated predictions as lying near the diagonal where predicted probability equals empirical accuracy. Points below the diagonal mean the model claims more confidence than it earns; points above mean it is underconfident. Annotate each point with its prediction count so readers can discount thin bins.

Expected calibration error. The guide defines ECE as the average gap between mean stated confidence and observed accuracy across probability buckets. We recommend weighting each gap by the share of predictions in its bin, so a sparse bin cannot dominate the score. ECE compresses the plot into one number for tracking, but it hides direction, so always publish it next to the plot.

Illustrative scenario: a made-up log of 2,000 decisions from one task type, binned into ten equal-width bins. The five lowest bins hold no predictions and are omitted.

Weighting each gap by its bin's share of the 2,000 predictions gives an ECE of about 0.047 in this invented example. Every observed accuracy sits below its mean confidence, so the plot would show consistent overconfidence, worst in the middle bins.

Bucket count choices. Ten bins is the guide's example, not a rule; the available material sets no universal bucket count, sample size or acceptable calibration error. More bins reveal finer structure but leave fewer predictions per bin and noisier accuracy estimates. Fewer bins are stable but can average away a miscalibrated region. Rerun the evaluation with a coarser and a finer bin count; if your conclusion flips, you do not yet have enough data.

Per task type. The guide notes that calibration can vary by task, so compute a separate table, plot and ECE for each question or decision type rather than one blended score.

Step-by-Step Guide

Step 1: Assemble the prediction log

Collect one row per decision with the decision itself, the stated probability, the task type, a timestamp and the observed outcome. Drop rows whose outcome is still unknown rather than guessing, because a confidence value cannot be validated from the model output alone. Record how the outcome was determined so a reviewer can audit the labels. Check that the probabilities you logged are the ones the model actually emitted, not values rounded or transformed downstream.

Pro tip: Log the raw probability at full precision; rounding to one decimal before binning quietly shifts predictions across bin edges.

Step 2: Split the log by task type

Group rows by question or decision type before any calculation. Calibration can differ across tasks, so a model that is well calibrated on routing tickets may be overconfident on extracting dates. Treat each task type as its own evaluation with its own table, plot and ECE. If a task type has too few rows to fill its bins, mark it as not yet evaluable instead of merging it into another.

Pro tip: Name task types by the decision the software makes, for example approve refund or route ticket, so the report maps directly onto automation rules.

Step 3: Choose a binning scheme and bin count

Decide between equal-width and equal-mass bins and pick a starting count, for example ten bins. Look at the histogram of stated probabilities first, since models often crowd their confidence near the top and leave low bins empty. Write the scheme and count into the report so later runs are comparable. Plan to rerun with a coarser and a finer count as a robustness check.

Pro tip: If most predictions fall above 0.8, consider narrower bins in that region, since that is where automation thresholds usually sit.

Step 4: Compute per-bin statistics

For each bin, count the predictions, average their stated probabilities and compute the fraction that were correct. Subtract to get the gap, keeping the sign so you know whether the bin is overconfident or underconfident. Flag bins whose count is too small to trust, using a minimum you set in advance, for example 50 predictions. Keep the full table; the summary score comes later.

Step 5: Draw the reliability plot

Plot mean confidence on the horizontal axis and observed accuracy on the vertical axis, with the diagonal drawn as the reference. Label each point with its prediction count or size the markers by count. Read the plot for patterns: a consistent sag below the diagonal signals overconfidence, while an S-shape suggests the model is too extreme at both ends. Mark flagged low-count bins visually so no one over-reads them.

Pro tip: Add error bars from a simple bootstrap over the rows in each bin; they make it obvious which gaps are noise.

Step 6: Calculate expected calibration error

Multiply each bin's absolute gap by its share of the total predictions and sum the results to get a weighted ECE. Report it alongside the plot and table, never on its own, because a single number hides whether the model errs high or low. Compare the score across bin counts to check stability. Track it per task type over time so drift after a model or data change shows up quickly.

Step 7: Report findings with caveats

Summarize each task type with its ECE, its most miscalibrated bins, the direction of the error and the sample size behind each claim. State plainly that calibration describes groups of predictions and says nothing certain about any single answer. Translate findings into operational terms, such as which confidence levels matched their observed accuracy well enough to consider for automation. List what you could not evaluate because of thin data.

Pro tip: Put the reliability plot first in the report; stakeholders grasp overconfidence from the picture faster than from the score.

Best Practices

• Record the outcome before judging anything. The practitioner guide treats logged predictions plus eventual true outcomes as the raw material of calibration, and without outcomes you are only describing the model's opinions about itself.
• Evaluate each task type on its own. Calibration can vary by task, so a blended score can look fine while one decision type is badly overconfident and another underconfident.
• Publish the reliability plot next to ECE. The score compresses direction away, and a reader needs the plot to see whether the model overclaims or underclaims and in which confidence range.
• Weight bin gaps by prediction share. An unweighted average lets a bin with a dozen predictions count as much as one with a thousand, which makes the score jump with noise rather than with real miscalibration.
• Test your conclusions against more than one bin count. Because there is no universal bucket count, a finding that only appears at one granularity is a sign of too little data, not a property of the model.
• Run the evaluation yourself even for models marketed as calibrated. An independent review found no public reliability curve or ECE figure for Jev, so your own measurements are the only evidence tied to your traffic.

Common Mistakes

• Reporting overall accuracy as if it showed calibration.: A model can be right often and still attach the wrong numbers to its answers. Compare confidence with observed accuracy inside each bin, which is what the practitioner guide prescribes.
• Pooling different task types into one calibration estimate.: Errors in opposite directions cancel out and hide real problems. Build a separate table, plot and ECE for each decision type.
• Drawing conclusions from sparse bins.: A bin with a few predictions can show a large gap purely by chance. Set a minimum count in advance, flag bins below it and add error bars before interpreting any gap.
• Treating a well-calibrated bin as a guarantee for individual answers.: Calibration describes a population, and Anthony Maio notes it does not tell you whether one particular answer is right. Use bin results to set policies, not to vouch for single decisions.
• Accepting vendor calibration claims in place of your own measurement.: Published calibration curves for Jev were not available when reviewers looked. Measure on your own logged outcomes before relying on any confidence level.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: Reinforcement Learning for Calibrated Decisions (RLCD)

Related Skills

• Calibrating Confidence to Outcomes
• Structuring Machine-to-Machine Decision Outputs
• Designing Schema-Constrained Decisions
• Handling Abstention and Uncertainty
• Designing Outcome-Based Reward Signals
• Estimating Decision Confidence

Sources

• RLCD explained: Reinforcement Learning for Calibrated Decisions
• Jev: The Language Model That Won't Talk - Anthony Maio
• Jev & System One Models: The Claim-vs-Evidence Guide
• Jev, three days in: what is known, what is guessed

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.