Marketing & Finance
A/B Test Calculator
Is variant B really better than A? Get conversion rates, uplift, p-value, a confidence interval and the sample size you need.
- Time: 3 minutes
- With: Marketing, product and growth
- You get: A decision: ship B, keep A or keep testing
How to use this worksheet
- Run the test for the planned duration, full weeks if behaviour varies by weekday.
- Enter visitors and conversions for both variants.
- Choose the confidence level you decided on before the test started (95% is the usual default).
- If the result is not significant, the note tells you roughly how many visitors per variant you would need.
Variant A (control)
The original version.
Variant B (challenger)
The version you want to test.
Write down where the numbers come from and what would change them.
Prompts and example
Example
Hypothesis: a shorter form increases sign-ups. Ran 14 days, traffic from paid search only.
Results
Before you finish
- The sample size and duration were fixed before the test.
- Both variants ran at the same time on comparable traffic.
- You are testing one change, not three.
How the test works
The calculator uses a two-sided z-test for the difference of two proportions with a pooled standard error. The p-value is the probability of seeing a difference at least this large if both variants were identical.
Stopping a test as soon as it looks significant inflates false positives. Decide the sample size up front and look at the result once.
Recommended reading
Werbung / Ad: Affiliate links. As an Amazon Associate I earn from qualifying purchases at no extra cost to you.
- Trustworthy Online Controlled ExperimentsRon Kohavi, Diane Tang, Ya Xu
The definitive guide to A/B testing from Microsoft's experimentation platform team. Covers statistics, pitfalls, and best practices.
- Testing Business IdeasDavid J. Bland & Alexander Osterwalder
Practical framework for designing and running experiments to validate business hypotheses and reduce risk.
- The Lean StartupEric Ries
Foundational book on validated learning and the Build-Measure-Learn loop through continuous experimentation.