Skill: formulating typed decision primitives for AI agents

Turn each fork in an agent into a typed question with a fixed answer space and a written rubric, so a decision model answers and code branches.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

A decision question is the unit of work you hand to a decision model instead of a text generator. In the split described on the Jev Engineering method page, the LLM writes, the decision model judges, and code acts on the judgement. This skill covers the middle piece: turning a fuzzy fork in an agent, such as "should we escalate this?", into a typed question that a model can answer and code can branch on without parsing prose.

The contract is narrow on purpose. The Made with Jev quickstart describes sending the text plus the questions you want answered and getting back a typed answer, a probability for every option and a confidence score, in 70 to 500 (source) milliseconds. That output is only as good as the question behind it. If two options overlap, probability splits between them and neither looks confident. If an option is missing, the model has to push a case into the wrong bucket, and nothing in the typed answer will warn you, because the answer is still perfectly well formed.

The skill has three parts. First, fix the answer space. LangChain's write-up on Jev as an evaluation judge frames this as a yes/no question for a binary decision, a complete list of options for a choice, or ordered levels for a score. Picking the shape early forces you to decide what the downstream code will actually do with each answer.

Second, write the rubric before you call the model. Akshay Pachaar's rollout notes recommend defining what belongs in every option up front, and starting with one bounded, low-risk decision with clear possible answers. A rubric written first stops you from quietly bending definitions to match whatever the model returned on its first few runs, which is how evaluation sets drift toward the model instead of toward the truth.

Third, keep questions atomic. One question asks one thing about one state. Multi-label problems, where several answers can be true at once, need a different shape than a single choice. A practitioner benchmark on dev.to reports that letting a choice decision compete first and then verifying with a noul decision lifted multi-skill routing from 9.0% to 81.0% (source) R@1 on its dataset.

The output is a small, versioned question set: each question with its type, its options or levels, and a rubric that a new teammate could apply by hand and reach the same answer the model should.

How It Works

Every decision question has three parts: the state it reads, the question text, and the answer space. The harness supplies the state. The answer space is what makes the question typed, because it is the only set of values the model is allowed to return and the only set your code has to handle. Akshay Pachaar's notes describe questions as the decisions you want the model to make about that state, which gives a quick test: if you cannot name the decision and the action each answer triggers, you do not have a question yet.

Three shapes cover most forks, following LangChain's framing of fixed answer spaces:

Yes/no fits gates, where one answer lets work proceed and the other stops or reroutes it. Choice fits routing, where exactly one of several paths should run. Ordered score fits cases where the levels mean something relative to each other, such as severity or relevance, and where code may later compare the score against a cutoff.

The reason answer spaces must be fixed and clean comes from how the output is consumed. The Made with Jev quickstart says the model returns a probability for every option. Code typically takes the top option and checks its probability. When two options describe overlapping situations, probability mass divides between them, and a correct but split answer looks uncertain. When the list is incomplete, cases that fit nothing still get assigned somewhere. The fix for the second problem is an explicit escape option, for example needs_human or none_apply, whose rubric entry says exactly when to use it.

The rubric is the written definition behind each option: what qualifies, what does not, and how to handle the common borderline. It is written before the first model call and then tested. The same notes recommend evaluation data with representative examples and expected answers, including ambiguous and adversarial cases. Ambiguous examples test whether your options really are mutually exclusive. Adversarial ones test whether the rubric holds when the input is written to mislead.

Multi-label decisions break the single-choice shape. If a task can need two skills at once, a choice question can only pick one, and a yes/no per candidate asked in isolation tends to say yes too often. For example, the dev.to benchmark recommends a two-stage shape: a choice question lets the candidates compete, then a verification question checks each of the leading candidates, which is the pattern behind its reported jump from 9.0% to 81.0% R@1. Setting the probability cutoffs that decide when to act is a separate job, covered in calibrating confidence thresholds and escalation paths.

Step-by-Step Guide

Step 1: Name the fork

Find a point in the agent where code branches on a judgement, and write the decision in one plain sentence. Next to it, list the action each outcome triggers downstream. If two outcomes trigger the same action, they are one outcome and should merge. If you cannot list actions at all, the fork is probably a text-generation task wearing a decision costume, and it belongs to the LLM.

Pro tip: Start with a bounded, low-risk fork, as the rollout notes from Akshay Pachaar suggest, so early mistakes are cheap.

Step 2: Pick the question type

Decide whether the fork is a gate, a route or a graded judgement. Gates become yes/no, routes become choice, and graded judgements become ordered scores. Resist turning a score into a choice with labels like low, medium and high unless code treats them as unrelated paths. If the answer can legitimately be several options at once, note that now, because it changes the shape in a later step.

Pro tip: Ask what the code does with the answer. Branching means yes/no or choice; comparing against a cutoff means score.

Step 3: Enumerate a complete answer space

Write every option the model may return, using short, stable identifiers that code can switch on. Check each pair for overlap: could a single input reasonably fit both? Then check coverage: is there an input from your sample that fits none? Add an explicit escape option for cases outside the list rather than hoping they never occur.

Pro tip: Use machine-friendly identifiers such as escalate_unknown and keep human wording in the rubric, so renaming a label never breaks code.

Step 4: Write the rubric before any model call

For each option, write what qualifies, what does not, and one borderline case with its correct answer. Keep it short enough that a teammate can apply it by hand in under a minute per input. Do this before running the model, because early outputs tend to anchor your definitions. The rubric becomes the reference for labelling examples and for reviewing disagreements later.

Step 5: Label a test set with hard cases

Pull real inputs from the decision point and label each with the answer the rubric gives. Deliberately include ambiguous inputs that sit between options and adversarial inputs written to mislead. Where two labellers disagree, fix the rubric, not the label. Only then run the question against the set and compare.

Pro tip: A useful starting size is, for example, a few dozen examples per option, weighted toward the borderline cases you actually see.

Step 6: Split multi-label decisions into choice then verify

If several answers can be true, do not force a single choice or fire off independent yes/no questions. Ask a choice question first so the candidates compete, then ask a verification question for each of the leading candidates. Keep both questions and their rubrics separate so you can test each stage on its own. Check whether the verified set matches your labels, not just whether the top choice does.

Step 7: Version the question set

Give each question an identifier and a version that changes whenever the text, options or rubric change. Store the rubric beside the question definition in code rather than in a separate document. Record the version with every decision so results can be traced to the exact wording that produced them. Replaying a fixed test set across versions is covered in the benchmarking skill.

Pro tip: Bump the version even for small wording edits; question phrasing can shift probabilities more than you expect.

Best Practices

• Map every option to a distinct downstream action. An option that no code path handles differently is noise, and it steals probability from options that matter.
• Keep one question per decision. Combining "is this safe and which worker should take it" into one choice multiplies the options and hides which half of the judgement went wrong.
• Always include an escape option with its own rubric entry. It gives out-of-scope inputs somewhere honest to land and gives you a signal for when the answer space needs a new option.
• Write rubrics as inclusion and exclusion criteria, not examples alone. Examples show the centre of a category; criteria define its edges, which is where decisions fail.
• Prefer ordered scores when levels are genuinely ordered. Code can then compare against a cutoff you tune later, instead of hard-coding which labels count as bad.
• Test the rubric on people before testing it on the model. If two teammates disagree on a label, the model has no stable target, and fixing the wording is cheaper than chasing model errors.

Common Mistakes

• Leaving options that overlap, such as billing and refund for a support router.: Rewrite options so every input fits exactly one, or split the question in two. Overlap divides probability between correct answers and makes good decisions look uncertain.
• Omitting a none-of-these option because the team believes the list is complete.: Add an explicit escape option and define it in the rubric. Without it, out-of-scope inputs are forced into a real category and the typed answer looks valid while being wrong.
• Writing or adjusting the rubric after seeing the model's first answers.: Freeze a rubric before the first call and label examples against it. Changes afterward should come from labeller disagreement, not from what the model happened to return.
• Treating a multi-label decision as a single choice.: Use a choice question to rank candidates, then verify each leading candidate with its own question. A single choice can only ever return one of several correct answers.
• Testing only clean, typical inputs.: Add ambiguous and adversarial examples to the test set. Typical inputs rarely expose a weak answer space; borderline and misleading ones do.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: Jev Engineering

Related Skills

• Benchmarking and Observing Agent Loops
• Separating Generation from Decision-Making
• Designing Routing and Ranking Policies
• Batching and Parallelizing Decisions
• Enforcing Deterministic Execution Boundaries
• Calibrating Confidence Thresholds and Escalation Paths
• Structuring Shared Agent State

Sources

• How to use Jev: first call in 5 minutes
• Akshay on X: https://t.co/haL8IGhx3h / X
• Jev is now available in LangSmith Evals - LangChain
• Benchmarking Jev: what a decision model can (and can't) do in an agent harness

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.