Skill: defining success criteria for AI agent verification

Write observable, thresholded acceptance criteria before an agent loop starts, so every verify step has a clear pass or fail.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

In the State-Questions-Action-Verify Loop, the verify step can only be as strict as the thing it checks against. This skill produces that reference point: a written acceptance specification that exists before the agent takes its first action, so the loop never has to guess whether it is finished.

The core move is describing an outcome instead of an activity. Guidance from Matter of Focus on robust success criteria says a criterion should describe what you want to know is happening, how you will know it is happening, what analysis will be needed, and the level of ambition. The Kentucky facilitator guide on success criteria makes the same point from a different field: criteria are observable outcomes, not a list of the activities performed. For an agent, 'ran the migration script' is an activity. 'Every row in the target table has a non-null customer ID' is an outcome.

The inputs are a small, fixed set. A governance guide to KPI definitions and success criteria lists the numeric target, the time window, the population in scope, how success will be verified, and any exceptions or assumptions. The GitLab handbook on verifiable outcomes adds a baseline, so you know where you started. In practice you gather the intended outcome, baseline, scope, time window, threshold, verification method, evidence source, tolerances, exceptions and disqualifying errors.

The output is a specification the loop can consume directly. For multi-step work, the Karpathy guidelines skill recommends stating a brief plan in which every step carries its own check, in the form 'step, then verify: check', and looping until verified. That structure also gives you a clean way to report partial progress: which checks have passed and which required checks remain open.

You will know this skill went wrong when a verifier returns 'looks done' with no artifact behind it, when two reviewers disagree about whether a criterion passed, or when nobody can find the data needed to evaluate it. Each of those symptoms points back to a criterion that was vague, uncollectable or written after the fact.

How It Works

A verifiable criterion is a statement about the world that can be checked without interpretation. TrueAbility's guide to performance-based assessment frames this as anchoring each item to a specific task or competency, and the same discipline applies to agents: each criterion should read as a state assertion that an automated system can query or a human can confirm without inference. If a reviewer has to ask 'what did they mean by complete?', the criterion is not finished.

Every strong criterion is assembled from the same components, which the governance guide to success criteria names as target, time window, population in scope, verification method, and exceptions or assumptions. Evidence is the sixth component: where the proof physically lives. The table below uses made-up example values to show the difference between a weak and a strong version of each.

The evidence column matters most for agents. The Kentucky facilitator guide asks for things someone will say, do, make or write that make the status visible; for software, the equivalent is a queryable system state, a test result or a saved artifact. Matter of Focus adds a practical filter: the evidence has to be readily collectable, either from an existing data source or through a practical way to create one. A criterion whose evidence cannot be gathered is a wish, not a check.

The verification questions are settled before any action runs. An INCOSE paper on verification requirements says a verification should answer its purpose, method, circumstances, environment and expected results. Answering those up front stops the agent, or the person reviewing it, from redefining success once results are in.

Not every outcome fits a single number. Six Sigma guidance on project success criteria recommends combining quantitative and qualitative measures in that case. A qualitative criterion still needs an observable condition, such as a rubric with named levels, so that a Score-style question or a human reviewer can apply it consistently.

Finally, the specification separates partial progress from success. Following the step-plus-check pattern in the Karpathy guidelines, each step has its own check, so the loop can report exactly which required checks pass and which remain incomplete instead of a single ambiguous status.

Step-by-Step Guide

Step 1: Name the intended outcome

Start from the business or user result the agent is supposed to change, not from the tasks it will perform. Six Sigma guidance recommends identifying the intended outcome first and only then translating it into measurable metrics and acceptance thresholds. Write one sentence that describes the world after success, in the present tense. Record the current baseline next to it so the target has a reference point.

If you cannot describe the end state without listing actions, the goal itself is still too vague to verify.

Pro tip: Ask 'what would I look at to know this worked?' and write down the answer as the outcome sentence.

Step 2: Rewrite the outcome as checkable assertions

Break the outcome into statements that are either true or false about the system or artifact. TrueAbility's assessment guide supports writing items that trace to a specific task, and each assertion should be something a script can query or a person can confirm without inference. Replace words like 'complete', 'clean' or 'works' with the concrete condition they stand for. Keep each assertion to one condition so that a failure points at one thing.

A good test is whether two reviewers would reach the same verdict independently.

Pro tip: Split any assertion containing 'and' into two; combined conditions hide which half failed.

Step 3: Set the threshold, time window and scope

For each assertion, decide the value that separates pass from fail, how long it must hold, and exactly which population it covers. The governance guide treats numeric target, time window and population in scope as required parts of a criterion. Name what is excluded as explicitly as what is included, such as segments, transaction types or file paths. For short agent tasks, the time window may simply be 'at the end of the run'; for deployed changes it may be a post-launch period.

Pick thresholds you are recommending, for example a 95% pass rate on a named test set, and note who chose them.

Step 4: Specify the verification method and evidence source

Write down precisely how each assertion will be checked: which query, test suite, sample or review step. The governance guide asks for the exact metric or data pull, the sampling approach, audit steps and required approvals. Then confirm the evidence is collectable, which Matter of Focus treats as part of a robust criterion. Name where the evidence will be stored so a later reviewer can find it.

If the only available evidence is the agent's own summary, find or build a better source before starting.

Pro tip: Run the verification method once against the current baseline; if it cannot produce a result today, it will not produce one after the agent runs.

Step 5: Attach a check to every step

Draft a short plan for the agent and pair each step with the check that proves it succeeded. The Karpathy guidelines use the pattern 'step, then verify: check' and summarise the practice as looping until verified. Step checks are intermediate evidence; the outcome assertions from earlier steps remain the final bar. Mark which checks are required for success and which are informational.

This lets the loop report partial progress precisely instead of claiming the whole task is done.

Pro tip: Keep step checks cheap and fast so the loop can run them every iteration without burning budget.

Step 6: Record exceptions and disqualifying errors

List the conditions under which a result should not be taken at face value. The governance guide calls for recording exceptions and assumptions such as outages, dependency failures or regulatory changes that trigger a review. Add disqualifying errors too: outcomes that fail the task even if every threshold passes, such as touching files outside scope. State what happens when an exception occurs, whether that is a pause, a rerun or a human review.

This keeps a lucky pass or an unlucky fail from being misread.

Pro tip: Write exceptions as triggers ('if X, then review'), not as excuses to explain away a failure afterwards.

Step 7: Trace, review and freeze the criteria

Check that every criterion maps back to the intended outcome or task scope. TrueAbility recommends tracing each item to the task or competency it measures, and criteria that do not map should be removed or revised. Review the draft with the people who own the outcome to confirm it is realistic, measurable and complete, as Six Sigma guidance suggests. Then freeze the specification and store it where the loop reads it.

Changing criteria mid-run should be a deliberate, recorded decision, not a quiet edit.

Pro tip: Version the specification alongside the agent's configuration so later runs can be compared against the same bar.

Best Practices

• Describe the end state, not the work. Kentucky's facilitator guide distinguishes observable outcomes from lesson activities, and agents need the same distinction because an action can run without producing its effect.
• Write every criterion before the first action. The INCOSE verification paper defines purpose, method, environment and expected results up front, which prevents success from being redefined to match whatever happened.
• Identify the evidence source while drafting, not after. Matter of Focus asks that evidence be readily collectable, and discovering a missing data source mid-loop usually means the loop cannot terminate cleanly.
• Give every criterion a threshold, a window and a scope. The governance guide treats these as required parts, and dropping any one of them lets a result pass on a convenient subset or a lucky moment.
• Pair quantitative checks with an explicit rubric when a number is not enough. Six Sigma guidance recommends mixing quantitative and qualitative measures, but the qualitative side still needs named levels so two reviewers agree.
• Keep required and informational checks separate. Following the step-plus-check pattern in the Karpathy guidelines lets the loop report partial progress honestly and stop only when every required check passes.

Common Mistakes

• Listing activities as criteria, such as 'run the tests' or 'update the docs'.: Completing an activity does not prove the outcome, as the Kentucky facilitator guide notes. Rewrite each as the observable result, for example 'all tests in the named suite pass on the final commit'.
• Using vague words like 'high quality', 'complete' or 'works' with nothing behind them.: Attach an observable condition, a threshold or a defined procedure to every such word, in line with TrueAbility's guidance on items that can be checked without inference. If you cannot, the word does not belong in the criterion.
• Writing criteria whose evidence nobody can actually collect or audit.: Name the data source, analysis and collection method while writing the criterion, as Matter of Focus and the governance guide recommend. Test the method against the baseline before the loop starts.
• Omitting scope and time window, so a pass on a small or friendly sample counts as success.: State the included and excluded population and how long the threshold must hold, both of which the governance guide lists as required. A result outside that scope or window is not a pass.
• Collapsing everything into one done/not-done flag.: Give each step its own check, following the Karpathy guidelines, and mark which are required. The loop can then show exactly what remains instead of oscillating between 'almost done' and 'failed'.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: State–Questions–Action–Verify Loop

Related Skills

• Recovering from Verification Failures
• Checking Post-Action Results
• Applying Loop Termination and Continuation Rules
• Formulating Typed Decision Questions
• Representing Current Agent State
• Selecting and Executing Bounded Actions

Sources

• Creating robust success criteria
• Precision English for Governance: Crafting KPI Definitions and
• karpathy-guidelines
• Success Plans | The GitLab Handbook
• Design Performance-Based Assessment Questions: 5-Step Guide
• [PDF] A Day in the Life of a Verification Requirement - MST.edu
• Project Success Criteria in Six Sigma: Best Practices &

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.