Question 1

What is statistical significance in an A/B test?

Accepted Answer

Statistical significance tells you how confident you can be that the difference in conversion rates between two variants is real and not due to random chance. The most common threshold is 95% confidence (p-value < 0.05), meaning there is less than a 5% chance the observed difference occurred by chance.

Question 2

How does this A/B test calculator work?

Accepted Answer

This calculator uses a two-proportion z-test. It computes the z-score from the difference in conversion rates divided by the pooled standard error, then calculates the two-tailed p-value. If the z-score exceeds 1.96 (p < 0.05), the result is statistically significant at 95% confidence.

Question 3

What sample size do I need for a valid A/B test?

Accepted Answer

As a rule of thumb, you need at least 100 conversions per variant for a meaningful result. Most A/B test calculators recommend running a sample size calculation before starting to avoid underpowered tests. Low sample sizes increase the risk of false positives.

Question 4

What does 'lift' mean in an A/B test?

Accepted Answer

Lift is the relative improvement of Variant B over Variant A, expressed as a percentage. For example, if Variant A converts at 5% and Variant B at 6%, the lift is 20%. A positive lift means B outperforms A, but always check if it is statistically significant before making decisions.

A/B Test Significance Calculator

📊 How it works

Know what to test next

Frequently Asked Questions

Related Tools

LTV Calculator

OKR Generator

Bug Priority Calculator