Interpreting Bayesian and Frequentist Experiment Results in PostHog
Teaches you how to read PostHog's experiment results dashboard, understand the difference between Bayesian credible intervals and frequentist p-values, and make confident ship-or-kill decisions based on the statistical output.
Open the experiment results page in PostHog and check the win probability for Bayesian tests or the p-value for frequentist tests. For Bayesian, a win probability above 95% (or below 5% for the inverse) signals a clear winner. For frequentist, a p-value below 0.05 with the minimum sample size reached indicates statistical significance. Always verify that the credible or confidence interval excludes zero before shipping a variant.
Outcome: You can read any PostHog experiment results dashboard, correctly interpret whether the data supports a winner, identify premature conclusions, and make a defensible ship-or-kill decision backed by the right statistical framework.
Prerequisites
- A running PostHog experiment with at least one goal metric collecting data
- Basic understanding of conversion rates and sample sizes
- Familiarity with the PostHog experiments tab (see sibling skill: Running A/B Tests in the PostHog Experiments Tab)
- Access to PostHog project with experiment results visible
Overview
Every experiment eventually reaches a moment where you need to stop watching the numbers climb and make a decision. Interpreting experiment results is the skill that separates teams who ship data-informed product changes from teams who either call experiments too early (wasting the statistical rigor they set up) or too late (leaving conversion gains on the table for weeks). PostHog surfaces both Bayesian and frequentist statistics on its experiment results page, and understanding what each number means, when to trust it, and when to wait is the core of this skill.
Inside the PostHog Experiments Onboarding A/B Test Method, this skill sits at the decision point between collecting data and shipping the winning variant. You have already designed your hypothesis, set up feature flags, segmented your cohorts, and launched the experiment. Now you are staring at a dashboard with win probabilities, credible intervals, conversion rates, and possibly p-values. The artifact you produce is a written interpretation memo: a short document (even a Slack message or Notion block) that states the metric, the observed lift, the statistical evidence, whether the result is conclusive, and the recommended action. This memo is what your team uses to decide whether to ship, extend, or kill the experiment.
The reason this skill requires its own focused practice is that PostHog gives you two statistical frameworks, Bayesian analysis (the default) and frequentist analysis, and they answer subtly different questions. Bayesian analysis tells you the probability that one variant is better than another given the data you have collected so far. Frequentist analysis tells you the probability of seeing data this extreme if there were truly no difference between variants. Mixing up these interpretations, or peeking at Bayesian results with a frequentist mindset, leads to premature calls and false confidence. By the end of this skill, you will know exactly which numbers to read, which to ignore at various stages, and how to document your conclusion so the team can act on it.
The posthog experiments docs describe these frameworks in detail, but the documentation assumes you already know which numbers matter for your decision. This skill fills that gap with a step-by-step reading process and worked examples across different experiment types.
How It Works
PostHog runs statistical analysis on your experiment data in real time, updating as new events arrive. The mental model you need is this: the dashboard is not a scoreboard showing who is winning. It is a probability calculator showing how confident you should be in a difference, given the data collected so far.
Bayesian analysis (PostHog's default) uses prior distributions and incoming data to compute a posterior distribution for each variant's conversion rate (or mean value for continuous metrics). From these posteriors, PostHog calculates a win probability: the percentage chance that variant B's true rate is higher than variant A's true rate. It also shows credible intervals, which are ranges that contain the true parameter value with a stated probability (typically 95%). The key insight is that Bayesian win probability is a direct answer to the question you actually care about: "Given what I have seen, how likely is it that B is genuinely better?" This makes it intuitive, but it also makes it tempting to peek early. A win probability of 80% after 50 users is not the same quality of evidence as 80% after 5,000 users, even though the number looks the same. The width of the credible interval is what tells you how much the estimate could still shift.
Frequentist analysis takes a different approach. It computes a test statistic (typically a z-test for proportions or a t-test for means), derives a p-value, and compares it to your significance threshold (alpha, usually 0.05). The p-value answers: "If there were truly no difference between variants, how likely would I be to see a result this extreme or more extreme?" A small p-value means the observed difference is unlikely to be pure noise. Frequentist analysis also produces confidence intervals, which are ranges that, if the experiment were repeated many times, would contain the true effect size 95% of the time. The critical discipline with frequentist tests is that you must reach your pre-calculated minimum sample size before checking the p-value for a final decision. Peeking inflates your false positive rate because every intermediate check is an implicit hypothesis test.
The practical difference: Bayesian results let you look at the dashboard any time and get a calibrated probability, but you should still wait until the credible interval is narrow enough to matter for your business. Frequentist results require discipline about when you look, but give you a framework with well-understood error rates. PostHog lets you choose which framework to use when you create the experiment, and you can see both if you switch modes, but the numbers mean different things and should not be compared directly.
The posthog experiments docs explain that the platform uses a Beta-Binomial model for conversion metrics in Bayesian mode and a standard two-proportion z-test in frequentist mode. Understanding this is not necessary for day-to-day interpretation, but it explains why Bayesian results update smoothly while frequentist p-values can jump erratically at low sample sizes. The Bayesian model is "smoothed" by its prior, which stabilizes early estimates but means early results are partially driven by the prior rather than your data. As sample size grows, the prior's influence shrinks and the posterior is dominated by observed data. This is why patience matters regardless of which framework you use.
Step-by-Step
Step 1: Open the experiment results page and confirm data is flowing
Navigate to the Experiments tab in PostHog, find your experiment, and click into it. Before reading any statistical output, verify that both variants are receiving traffic. Check the participant count for control and test. If one variant has zero or near-zero participants, you have a feature flag configuration issue, not a results interpretation task.
Confirm that the primary goal metric (the one you defined in your hypothesis) is recording events for both variants. Look at the event count, not the conversion rate, at this stage. If events are flowing for both variants, proceed. If not, troubleshoot the flag assignment or event instrumentation before interpreting any numbers.
Tip: PostHog shows a banner when an experiment has not yet reached the minimum recommended sample size. If you see this banner, you can observe trends but should not make a ship decision.
Step 2: Identify which statistical framework your experiment is using
Check whether the experiment was configured for Bayesian or frequentist analysis. This is set at experiment creation time and determines which numbers on the results page are meaningful. In Bayesian mode, the primary output is a win probability and credible interval. In frequentist mode, the primary output is a p-value and confidence interval.
If you are unsure which mode was chosen, look at the results card: Bayesian mode shows "Probability of being best" as a percentage, while frequentist mode shows a p-value as a decimal. Write down which mode you are interpreting so you do not accidentally apply the wrong reading framework in the next steps.
Tip: If your team did not consciously choose a framework, PostHog defaults to Bayesian. This is fine for most onboarding experiments because it lets you monitor results continuously without inflating false positive rates.
Step 3: Read the primary metric result for Bayesian experiments
For Bayesian experiments, locate the "Probability of being best" for each variant. This is the win probability. A 95% probability for the test variant means, given the observed data, there is a 95% chance the test variant's true conversion rate is higher than control's. Next, look at the credible interval for the difference between variants.
4%". This means the true lift is 95% likely to fall somewhere in that range. 2%"), the data has not yet ruled out the possibility of no effect or a negative effect. Record three numbers: the win probability, the lower bound of the credible interval, and the upper bound.
These three numbers are what your interpretation memo will reference.
Tip: A win probability of 90% with a credible interval of +0.1% to +12% is much weaker evidence than a win probability of 90% with a credible interval of +2.5% to +5.1%. The width of the interval tells you how uncertain the estimate still is.
Step 4: Read the primary metric result for frequentist experiments
For frequentist experiments, locate the p-value. 05 (or whatever alpha you set at experiment creation), the result is statistically significant. Next, check the confidence interval for the difference. Like the Bayesian credible interval, this is a range for the estimated lift.
If the confidence interval does not include zero, the result is significant at your chosen level. Crucially, verify that the experiment has reached the minimum sample size PostHog calculated when the experiment was created. If you have not reached that sample size, the p-value is unreliable regardless of how small it looks. Record: the p-value, the confidence interval bounds, the current sample size, and the target sample size.
Tip: If you are tempted to peek at frequentist results before minimum sample size, consider switching your mental model to Bayesian. PostHog lets you view Bayesian stats on any experiment. Use Bayesian for monitoring progress, but only rely on the frequentist conclusion once sample size is met.
Step 5: Check secondary metrics for guardrail violations
Most experiments track more than one metric. Your primary metric is the one you are trying to improve (for example, onboarding completion rate). Secondary metrics are guardrails: metrics you need to not get worse (for example, session duration, or retention at day 7). Scroll down to secondary metrics on the results page.
For each guardrail metric, check whether the test variant shows a statistically significant negative effect. A small, non-significant dip in a guardrail is usually acceptable. A significant decline, especially if the credible or confidence interval for the guardrail excludes zero in the negative direction, is a red flag that the test variant may be improving conversions at the expense of something important. Document any guardrail that shows a concerning movement, even if it has not reached significance yet.
Tip: Onboarding experiments often improve immediate completion rates while reducing downstream engagement. Always check a retention or activation metric at least 7 days out as a guardrail.
Step 6: Assess practical significance alongside statistical significance
Statistical significance tells you the effect is real. Practical significance tells you the effect matters. 3% may be statistically significant with a large sample, but may not justify the engineering and design cost of shipping and maintaining the change. Before writing your interpretation memo, define the minimum effect size that would make the experiment worth shipping.
For most onboarding flows, a lift below 1 percentage point on a key conversion step is marginal. A lift above 3 percentage points is clearly actionable. If the observed lift falls in between, use the credible or confidence interval: if the lower bound is above your minimum meaningful effect, the result is both statistically and practically significant. If the lower bound is below your minimum, the true effect could be too small to matter even though the point estimate looks good.
Tip: Write down your minimum meaningful effect size before looking at the results. This prevents you from rationalizing a small lift as "good enough" after the fact.
Step 7: Decide: ship, extend, or kill
Based on your readings, make one of three decisions. 05 with minimum sample in frequentist), the credible or confidence interval excludes zero, the practical effect size exceeds your minimum, and no guardrail metrics show significant degradation. Extend the experiment if the win probability is between 80% and 95% (Bayesian) or you have not yet reached minimum sample size (frequentist), and the directional trend is positive. Set a specific end date for the extension, typically 1-2 more weeks, and define what you need to see to ship.
Kill the experiment if the control variant is winning with high probability, or if guardrail metrics are degrading significantly, or if the experiment has run for more than 2x the expected duration with no clear signal.
Tip: Never leave an experiment running indefinitely in "let's see" mode. Set a hard stop date when you extend, and commit to making a decision on that date regardless of where the numbers land.
Step 8: Write the interpretation memo
Document your conclusion in a structured format that your team can act on. Include: the experiment name and date range, the primary metric name and observed lift (point estimate plus interval), the statistical evidence (win probability or p-value), the sample sizes for each variant, the practical significance assessment, the guardrail metric status, and the decision (ship, extend, or kill) with reasoning. Keep it to one page or one Slack message. This memo becomes the artifact that justifies the next action in the workflow, whether that is shipping the winning variant using the cleanup process or archiving the experiment as a non-result.
Share it with stakeholders before acting on the decision.
Tip: Include a screenshot of the PostHog results dashboard in the memo. Numbers in context are easier for non-technical stakeholders to trust than numbers alone.
Examples
Example: Onboarding tooltip experiment for a B2B SaaS tool (Bayesian, small team)
A 5-person product team at a B2B project management tool tested whether adding contextual tooltips to the onboarding checklist would increase checklist completion from 34% to 40%. They used PostHog's Bayesian mode, targeting new signups only. After 3 weeks, they had 1,200 users per variant. The primary metric was checklist completion rate. The guardrail metric was 7-day retention.
2% for the tooltip variant. 8 percentage points. 1%, meaning the interval excluded zero. The guardrail metric, 7-day retention, showed a win probability of 62% for the test variant (essentially neutral, no degradation).
The team had set a minimum meaningful effect of 2 percentage points before the experiment started. 4%) was below the 2pp minimum, which gave the PM slight pause. 8pp was well above the threshold, and the credible interval was still moderately wide because of the sample size. They decided to extend the experiment one more week.
8%. They wrote the interpretation memo, recommended shipping, and handed off to the engineer to follow the variant cleanup process.
Example: Pricing page CTA experiment for a B2C subscription app (frequentist, high traffic)
A consumer subscription app with 50,000 new signups per month ran a frequentist experiment testing two CTA button texts on the pricing page shown during onboarding. The minimum detectable effect was calculated at 1.5pp, requiring 8,400 users per variant. The primary metric was subscription start rate. The guardrail was bounce rate on the pricing page.
After 10 days, the experiment reached 9,100 users per variant, surpassing the minimum sample. The team opened PostHog and checked the frequentist results. 05 threshold. 2% subscription start rate).
8%. The confidence interval excluded zero, confirming significance. 72). 1pp was above, and they had pre-committed to using point estimate for the practical threshold check), and clean guardrails, they wrote the memo and recommended shipping.
5pp practical threshold, leading to a brief discussion. They decided the evidence was strong enough given the cost of the change was negligible (a single text string), and shipped the variant.
Example: Onboarding flow redesign experiment (Bayesian, inconclusive result)
A mid-stage startup redesigned their entire 5-step onboarding flow and ran a Bayesian experiment comparing the old flow (control) to the new flow (test). They targeted 2,000 users per variant over 4 weeks. The primary metric was activation (completing at least one core workflow within 48 hours of signup). Guardrails were support ticket creation rate and 14-day retention.
After 4 weeks, the experiment had 2,100 users per variant. 4%. The interval included zero, meaning the data had not ruled out that the new flow was actually worse. 7 percentage points.
3% for ticket creation rate. Fourteen-day retention was neutral at 54% win probability. The team wrote a memo documenting the inconclusive primary result and the potentially negative guardrail signal. Their decision: kill the experiment and not ship the redesign as-is.
The interpretation was that the new flow might produce a small conversion lift, but the effect was too small to detect confidently at their traffic level, and the support ticket increase suggested the new flow introduced confusion. They extracted the two individual steps from the redesign that user session recordings suggested were most effective, and planned two smaller, more focused experiments to test each step independently.
Example: Multi-variant onboarding email sequence test (Bayesian, B2B, three variants)
A developer tools company ran an A/B/N experiment with three email sequence variants for post-signup onboarding: Variant A (existing 5-email sequence, control), Variant B (3-email sequence, shorter and more actionable), and Variant C (5-email sequence with personalized content based on signup role). They used Bayesian mode and ran for 6 weeks, collecting 3,400 users across the three variants. The primary metric was "first project created within 7 days."
The results page showed three variants with their respective win probabilities. Variant B (short sequence) had a 14% probability of being best. Variant C (personalized) had an 82% probability of being best. Control had a 4% probability of being best.
9%, excluding zero but with a wide range. 4%, solidly including zero. The team noted that Variant C was promising but had not crossed the 95% threshold. They decided to extend the experiment for 2 more weeks with a hard cutoff.
8%. This was close to but did not reach 95%. The team discussed the tradeoff: the personalized sequence required ongoing maintenance of role-based content, which had a real engineering cost. At 91% probability, there was roughly a 1-in-11 chance the variant was not actually better.
They decided to ship Variant C but scheduled a follow-up check at 12 weeks to verify the lift held with a larger sample in production. The memo documented the 91% threshold decision, the reasoning for accepting slightly higher uncertainty, and the follow-up plan.
Best Practices
Define your decision criteria before the experiment collects data: specify the significance threshold (95% win probability or p < 0.05), the minimum meaningful effect size, and the guardrail metrics you will check. Writing these down in the experiment description inside PostHog prevents post-hoc rationalization when the results are ambiguous. Teams that skip this step tend to move the goalposts once they see numbers they like or dislike.
Use Bayesian mode for most onboarding experiments because it lets you monitor results at any time without inflating error rates. Frequentist mode is appropriate when you need a fixed-sample test with well-characterized Type I error rates, which is common in regulated or high-stakes environments. For typical SaaS onboarding experiments, Bayesian provides the flexibility you need since onboarding flows have variable traffic and you will want to check progress regularly.
Wait for the credible interval to narrow to a range your business can act on, not just for the win probability to cross 95%. A win probability of 96% with a credible interval of +0.2% to +15% tells you the variant is probably better, but you have no idea by how much. If the lower bound of +0.2% is below your minimum meaningful effect, the uncertainty is still too high to make a good decision. Let the experiment run until the interval is tight enough to be useful.
Check all goal metrics, not just the primary one. PostHog lets you attach multiple metrics to an experiment, and a variant that lifts one metric while tanking another is not a winner. Review every metric before writing your interpretation memo. Guardrail violations on secondary metrics are the most common reason an apparently successful experiment should actually be killed.
Never compare Bayesian and frequentist numbers side by side as though they measure the same thing. 05. They answer different questions using different math. Pick one framework for your final decision and reference only that framework's numbers in your memo.
Mixing frameworks creates confusion for stakeholders and can lead to contradictory conclusions.
Run experiments for complete business cycles. If your onboarding flow gets different traffic on weekdays versus weekends (nearly all SaaS products do), run the experiment for full weeks, not partial weeks. A result that looks significant on Wednesday may reverse by Sunday because the weekend cohort behaves differently. Minimum recommended duration is 2 full weeks for most onboarding experiments.
Document every experiment result, including non-results and losses. Negative results and inconclusive experiments are valuable data. They prevent your team from re-running the same test six months later, and they build institutional knowledge about what does and does not move your onboarding metrics. Store memos in a shared location with tags for the metric, the variant type, and the outcome.
Common Mistakes
Calling a winner based on win probability alone when the credible interval still includes zero
Correction
3%. This means the model thinks the variant is probably better, but it has not ruled out the possibility that it is actually slightly worse. The tell is the lower bound of the credible interval crossing zero. When you see this, the experiment needs more data.
Wait until both the win probability exceeds your threshold AND the credible interval excludes zero before calling a winner.
Peeking at frequentist p-values before reaching minimum sample size and acting on a significant result
Correction
Frequentist tests produce valid p-values only at the pre-specified sample size. Checking the p-value after 500 users when you planned for 5,000 is effectively running 10 hypothesis tests, which inflates your false positive rate dramatically. 41 at 5,000. The early significance was noise.
If you need to monitor results before reaching sample size, switch to viewing Bayesian statistics for progress monitoring and only use the frequentist conclusion once the sample target is hit.
Ignoring practical significance because the statistical result looks clean
Correction
1% lift in conversion rate can become statistically significant. 1% lift introduces code complexity, design debt, and maintenance cost that likely exceeds the value. This happens most often on high-traffic pages where PostHog accumulates large samples quickly. Before writing your memo, always check the point estimate of the lift against your minimum meaningful effect size.
If the upper bound of the credible interval barely clears your minimum, the experiment is telling you the effect is real but too small to bother with.
Treating the experiment as failed because the test variant lost
Correction
A test variant that clearly loses is a successful experiment. You learned that the change makes things worse, which is valuable information. The real failure mode is an inconclusive experiment that runs for weeks and produces a win probability hovering around 50-60%. This typically means the effect size is too small to detect at your traffic level, or the variants are not different enough.
When a variant loses decisively (control win probability > 95%), document what you learned about why the change hurt, kill the experiment, and move on. These insights often inform the next experiment's hypothesis.
Switching between Bayesian and frequentist frameworks mid-experiment to find the one that gives a significant result
Correction
Both frameworks analyze the same data but with different assumptions and different thresholds. It is always possible that one shows significance while the other does not, especially near boundary cases. If you find yourself checking both and picking the one that agrees with your desired outcome, you are p-hacking. Choose one framework at experiment creation time, commit to it in your hypothesis document, and reference only that framework's output in your interpretation memo.
The framework choice should be documented before any data is collected.
Running the experiment indefinitely because the results are "almost" significant
Correction
Experiments that hover near the significance boundary for more than 2x the expected duration are telling you something: the effect is probably smaller than you hypothesized. Continuing to run them wastes the opportunity cost of testing something else. Set a hard maximum runtime (typically 4-6 weeks for onboarding experiments) and commit to making a decision at that point. If the result is still ambiguous, the correct interpretation is: the effect, if it exists, is too small to detect at your traffic level, and you should move on to higher-impact experiments.
Other Skills in This Method
Running A/B Tests in the PostHog Experiments Tab
Step-by-step walkthrough of creating, launching, and monitoring an A/B test using PostHog's Experiments UI, including variant allocation and goal setup.
Setting Up PostHog Feature Flags for Experiment Variants
How to create and configure feature flags in PostHog to assign users to control and test variants in an A/B experiment.
Comparing PostHog Experiments with Eppo, LaunchDarkly, and Other Platforms
How to evaluate PostHog's experimentation capabilities against dedicated tools like Eppo, Statsig, and LaunchDarkly based on analysis methods, integrations, and pricing.
Shipping the Winning Variant and Cleaning Up Feature Flags
How to roll out the winning experiment variant to 100% of users, remove the losing variant's code, and archive feature flags to keep your codebase clean after an experiment concludes.
Designing Experiment Hypotheses and Success Metrics for Onboarding
How to formulate a clear hypothesis, choose primary and secondary conversion metrics, and define what winning looks like before launching an onboarding A/B test.
Segmenting New User Cohorts for Onboarding Experiments
How to target experiments specifically to new users or sign-up cohorts using PostHog's person properties and cohort filters to avoid contaminating results with existing users.
Integrating PostHog A/B Tests with Webflow and Marketing Pages
How to implement PostHog experiments on no-code or marketing landing pages using the JavaScript snippet, Webflow custom code, and anti-flicker techniques.
Frequently Asked Questions
How do I interpret PostHog experiment results when Bayesian win probability is between 80% and 95%?
A win probability between 80% and 95% means the data leans toward a winner but has not reached conventional confidence. Check the credible interval: if it excludes zero and the lower bound exceeds your minimum meaningful effect, you may have a case for shipping, especially if the change is low-cost to maintain. If the interval still includes zero, extend the experiment with a hard cutoff date. Document the current state and set specific criteria for what must be true at the cutoff to ship. Many teams treat 90% as an acceptable threshold for low-risk changes and 95% for changes with significant maintenance costs.
Should I use Bayesian or frequentist mode for my PostHog experiment?
Use Bayesian for most product experiments, especially onboarding flows where you want to monitor results continuously. Bayesian analysis produces valid probabilities at any sample size, so peeking does not inflate error rates. Use frequentist when you need a fixed-sample design with well-characterized false positive rates, such as experiments with external stakeholder reporting requirements or regulated contexts. PostHog defaults to Bayesian, and the posthog experiments docs recommend it for most use cases. The key discipline difference: Bayesian lets you look anytime but you still need a narrow credible interval to act, while frequentist demands you wait for minimum sample size before making any conclusion.
How long should I run a PostHog experiment before interpreting results?
At minimum, run the experiment for 2 full weeks to capture weekday and weekend traffic patterns. PostHog calculates a recommended minimum sample size based on your expected effect size and baseline conversion rate. Do not interpret final results until you reach that sample size for frequentist tests. For Bayesian tests, focus on the credible interval width rather than a sample target: if the interval is still wide enough that the upper and lower bounds would lead to different business decisions, the experiment needs more data. Set a maximum duration of 4-6 weeks for most onboarding experiments. If the experiment has not reached significance by then, the effect is likely too small to detect at your traffic volume.
Why does my PostHog experiment show a high win probability but a credible interval that includes zero?
This happens when the posterior distributions for the two variants overlap slightly but the majority of the probability mass favors one variant. 2% means the model thinks the test is probably better, but there is still a meaningful chance (roughly 12%) that it is equal or slightly worse. The credible interval including zero is the stronger signal here: it means you have not yet ruled out no effect. This is most common at moderate sample sizes where the trend is real but the data has not yet pinched the interval tight enough. Wait for more data rather than acting on the win probability alone.
How do I handle conflicting results between primary and guardrail metrics?
When the primary metric improves but a guardrail metric degrades, you have a tradeoff, not a clear winner. First, verify the guardrail degradation is statistically meaningful (not just directional noise). 1, take it seriously. Then assess the magnitude: a 5% improvement in onboarding completion with a 1% decrease in 7-day retention might be acceptable, but a 5% improvement with a 10% retention drop is not. Document both effects in your memo and present the tradeoff to stakeholders explicitly. In most onboarding experiments, retention and engagement guardrails should take precedence because short-term conversion gains that reduce long-term engagement destroy more value than they create.
Can I restart a PostHog experiment if the initial results look wrong?
You can, but you should not restart an experiment to get different numbers. Legitimate reasons to restart include: discovering a bug in event tracking that corrupted data, finding that the feature flag was not assigning users correctly, or realizing you targeted the wrong user segment. In these cases, fix the issue, create a new experiment (do not reuse the old one, since its data is tainted), and start fresh. If you restart because you did not like the results, you are introducing selection bias and your next result will be untrustworthy. Document why you restarted and ensure the new experiment has a clean hypothesis and clean data.
How do I interpret results for experiments with very low traffic where PostHog shows wide credible intervals after weeks?
Low-traffic experiments require larger effect sizes to detect reliably. If your credible interval is still spanning 15+ percentage points after 3-4 weeks, the experiment is underpowered for the effect you are trying to detect. You have three options: increase the effect by making bolder variant changes (a bigger difference between control and test), broaden the audience (segment less aggressively to include more users), or accept that you need a longer experiment runtime. For onboarding flows with under 200 signups per week per variant, consider testing only high-impact changes where you expect a lift of 5+ percentage points. Small, incremental optimizations are not testable at low traffic volumes. The posthog experiments docs recommend a minimum of 100 unique users per variant before drawing any conclusions, even directional ones.