Statistical Significance
The probability that observed differences between variants are real rather than random chance.
1 min readLast updated Apr 2026
Quick Reference
CategoryConversion Rate Optimization
Related Terms2
The probability that observed differences between variants are real rather than random chance.
Why It Matters
Without statistical significance, you're making decisions based on noise. A variant might appear 5% better, but if that difference isn't significant, it could easily flip to 5% worse with more data. Significance tells you when to trust results.
Practical Example
Scenario
A pet food brand's test shows Variant B converting at 4.2% vs Variant A at 3.8%. Looks like a winner.
Calculation
But with only 500 conversions, the confidence level is 78%—well below the 95% thresholdResult
They continue the test. After 2,000 conversions, the difference shrinks to 4.0% vs 3.9% with 62% confidence. The apparent winner was just random variation.
Pro Tips
- 1Always aim for 95% confidence minimum (99% for major changes)
- 2Use a sample size calculator before starting to know how long to run the test
- 3Don't peek at results and stop early when one variant looks ahead
- 4Consider both statistical and practical significance—a 0.1% lift may not be worth implementing
Common Mistakes to Avoid
Declaring winners based on early results before reaching significance
Using 90% confidence or lower, which means 1-in-10 chance results are random
Ignoring confidence intervals—a 'winner' with wide intervals may still lose