Designing Validated Learning Experiments

Design validated learning experiments: pick the test, from landing page to concierge MVP or Wizard of Oz test, and set pass marks before any data arrives.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

Validated learning experiments are small, deliberate tests that turn one assumption into evidence. Eric Ries calls validated learning "a rigorous method for demonstrating progress when one is embedded in the soil of extreme uncertainty" (The Lean Startup principles). An experiment earns that name when it is designed before it runs: the hypothesis, the test, the metric and the criteria for passing and failing are all written down first. This skill covers how to design product experiments that meet that standard.

Hypothesis-driven product development depends on this discipline. Without it, a team launches something, looks at the numbers, and finds a story that fits what it already wanted to do. With it, the team agrees in advance what result would change its plan. The Lean Startup method explains how experiments feed the Build-Measure-Learn loop. This page covers the design of a single experiment.

Lean startup experiments come in many forms. A landing page test measures whether people will sign up or pay for something described but not yet built. A concierge MVP delivers the service by hand to a few customers. A Wizard of Oz test presents what looks like a working product while people do the work behind the scenes. A split test compares two versions of a live product. Each answers a different kind of question at a different cost, and choosing well is half of the design.

Good design also means knowing when to stop. Research summarized in The Conversation cites a study finding that more validation is not necessarily better and a diminishing and negative relationship between the number of validated hypotheses and team success. Experiments are for the assumptions that matter. Testing everything wastes the time that the method is meant to save.

The output is an experiment brief that anyone on the team can read: what we believe, how we will test it, what we will measure and what result counts as right. When the result comes in, the brief makes the decision almost mechanical.

How It Works

Strategyzer's Test Card is a compact template for the brief. It has four lines: "We believe that" for the hypothesis, "To verify that, we will" for the test, "And measure" for the metric, and "We are right if" for the criteria. It also asks the team to rate how critical the hypothesis is, the test cost and the data reliability (Strategyzer Test Card). Those three ratings are the core trade-off in experiment design: test the critical assumptions first, and prefer tests that give reliable data for little cost.

The test type follows from the question. Ries plans in reverse, starting from what the team needs to learn, then what to measure, then what to build (Ries on planning in reverse). A demand question can often be answered with a landing page test MVP or an ad smoke test; Ries notes that "a simple AdWords smoke test" would have revealed how bad one feature concept was (Ries, MVP guide). A question about whether the service creates value needs customers to actually receive it, which a concierge MVP or a Wizard of Oz test provides.

The concierge MVP and the Wizard of Oz test differ in what the customer knows. OpenClassrooms describes the concierge MVP as "manually helping your users accomplish their goals," so the customer sees the human effort, while a Wizard of Oz MVP "gives a certain impression of your solution from the outside" while the inner workings are something else (OpenClassrooms, 4 types of MVP). The Wizard of Oz name comes from human-computer interaction research, where John F. Kelley used it for studies in which a hidden person operates a system the participant believes is automated (Wikipedia, Wizard of Oz experiment).

Measure behavior. Sign-ups, payments, repeat use and referrals are actions with a cost to the customer. Survey answers about future intentions cost nothing and are much weaker evidence. When comparing versions, a split test is the most direct design, but it must run both versions at the same time with random assignment. The Wikipedia article on lean startup explains that a serial test, with one version this week and another next week, lets outside events change behavior between the two periods.

Criteria are set before data arrives. Tristan Kromer recommends that each criterion name a specific impact, a timeframe and a probability level, and that criteria be written before any data is collected to counter cognitive bias (Kromer). Write both a pass line and a fail line. A result between them is a legitimate outcome that calls for a sharper follow-up test.

Step-by-Step Guide

Step 1: Pick the assumption worth testing

Start from the ranked list of leap-of-faith assumptions and choose one that is critical to the business and weakly supported by evidence. Skip assumptions you could not act on whatever the result. Write down what decision the experiment will inform, such as whether to build a feature or target a segment. If no decision depends on the answer, do not run the experiment.

Step 2: Write the hypothesis

State the assumption as a falsifiable claim about a specific customer and a specific behavior, using the lean startup hypothesis template. Name who, what they will do and in what setting. Avoid words like "like" or "interested", which cannot be observed. The hypothesis should make a clear prediction that a result could contradict.

Step 3: Choose the cheapest adequate test

List the test types that could produce evidence on the hypothesis: interviews, landing page test, smoke test, concierge MVP, Wizard of Oz test, prototype test or split test. Rate each on cost and on how reliable its data would be, as the Test Card suggests. Pick the cheapest test whose data would be strong enough to change your decision. Move to more expensive tests only once cheaper ones have passed.

Step 4: Define the metric and the sample

Choose one primary behavioral metric that directly reflects the hypothesis, plus any guardrail metrics that would reveal a misleading result. Define the audience, how they will be recruited and how many you need to see a meaningful difference. If you are comparing versions, randomize who sees which, and run both at once. Write down how each customer will be counted.

Step 5: Set pass and fail criteria in advance

Write a pass line and a fail line for the primary metric, each with a timeframe. State what you will do in each case, and what you will do if the result falls between the lines. Share the criteria with the people who will make the decision before the test starts. This is the step that makes the learning validated.

Step 6: Run the test without moving the goalposts

Launch to the defined audience and monitor that data is being recorded. Do not change the test or the criteria mid-run. If something breaks the test, stop, fix it and restart with a fresh sample. Keep notes on anything unexpected, since surprises are often worth their own experiment.

Step 7: Read the result and record the learning

Compare the result with the pre-set criteria and record whether the hypothesis passed, failed or landed between. Add what customers said or did that explains the result. Update the assumption list and pick the next experiment, or bring the evidence to the next pivot or persevere decision.

Best Practices

• Test one assumption per experiment. When a test covers two, a failure cannot tell you which one was wrong.
• Prefer behavior that costs the customer something, such as time, money or reputation. It predicts future behavior far better than stated intent.
• Run split tests in parallel with random assignment. A serial comparison mixes the effect of your change with whatever else changed that week, as the Wikipedia summary explains.
• Start cheap and escalate. A landing page or smoke test can kill a weak idea before a concierge MVP is worth the effort.
• Write the brief where the whole team can see it. Shared criteria make it harder to reinterpret a result after the fact.
• Stop testing assumptions that no longer matter. The research cited in The Conversation suggests over-validation can hurt.

Common Mistakes

• Setting the criteria after seeing the data: Any result can be framed as a win once you know it. Write pass and fail lines before launch.
• Measuring opinions: Survey answers about future use feel like data but rarely predict it. Measure what people do.
• Running tests too small to read: A handful of visitors cannot distinguish a real effect from noise. Size the audience to the difference you need to detect, or pick a test with a stronger signal.
• Confusing a concierge test with a Wizard of Oz test: In a concierge test the customer knows a person is helping, so it tests the value of the result. To learn how customers react to what looks like the finished, automated product, run a Wizard of Oz test instead.
• Testing what is easy instead of what is risky: Button colors are easy to test and rarely decide the fate of a product. Start with the assumption that would hurt most to get wrong.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: Lean Startup

Related Skills

• Lean Startup Hypothesis Template: Testable Hypotheses
• Types of MVP: How to Choose the Right Format
• How to Build a Minimum Viable Product (MVP)
• Running the Build-Measure-Learn Loop
• Innovation Accounting Metrics: Tracking Real Progress
• Pivot or Persevere: When to Pivot a Startup
• Customer Discovery Interview Questions and Technique

Sources

• The Lean Startup: Principles
• Strategyzer: The Test Card
• Eric Ries quote on planning the loop in reverse
• Eric Ries: Minimum Viable Product, a guide
• OpenClassrooms: The 4 types of minimum viable product
• Wikipedia: Wizard of Oz experiment
• Wikipedia: Lean startup
• Tristan Kromer: Pivot or persevere decision
• Ratinho and Bruneel: Is the lean start-up method really universally applicable?

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.