Reading results

We use fixed-horizon statistics, as the CRO playbook recommends:

• Before launch we compute the sample size per variant from your baseline rate and the minimum detectable effect (MDE), at 95% significance and 80% power.
• No decision before the planned sample is reached and the test has run at least two full weeks (14 days by default; 7 is the shortest you can choose in Settings). Peeking and stopping early inflates false positives.
• Winner = p < 0.05 (Bonferroni-corrected for A/B/n), the 95% confidence interval of the lift is entirely above zero, and no guardrail is harmed (purchases, AOV, revenue per visitor).
• Inconclusive means the test couldn't prove a difference. It does not mean the change does nothing.
• Stopped for harm: after half the planned sample, if the treatment is almost certainly worse (P(better) < 1%), we stop and restore the original.
• Sample-ratio mismatch (SRM): if visitors aren't split as planned (p < 0.001), the test is paused, because something is broken.
• Revenue per visitor can be the primary metric. It's tested with a Welch t-test on per-visitor revenue, and the plan accounts for revenue's high variance, so it needs more traffic than a conversion test.
• Secondary metrics and device breakdown explain a result; they never decide it. Winning on mobile but losing overall is not a win.