Conversion Rate Optimization

Sample Size

The number of visitors or conversions needed to reach statistical significance.

1 min readLast updated Apr 2026

The number of visitors or conversions needed to reach statistical significance.

Why It Matters

Testing with too few visitors produces unreliable results—you might implement a change that actually hurts conversion. Calculate required sample size before testing to know if a test is even viable for your traffic levels.

Practical Example

Scenario

A home goods brand wants to test a checkout redesign. Current conversion: 2.5%. They want to detect a 10% relative improvement (to 2.75%).

Calculation

Sample size calculator shows: 15,000 visitors per variant needed for 95% confidence, 80% power

Result

With 40,000 monthly checkout visitors, they can run this test in ~3-4 weeks. A smaller test detecting a 5% improvement would need 60,000 per variant—3+ months.

Pro Tips

  • 1Calculate sample size before any test—use free calculators like Evan Miller's or Optimizely's
  • 2Smaller expected effects require larger samples (detecting 2% lift needs 10x the sample of detecting 20% lift)
  • 3Focus on high-traffic pages where you can reach significance faster
  • 4Consider testing macro-conversions (purchases) vs micro-conversions (add-to-cart) based on volume

Common Mistakes to Avoid

Starting tests without calculating required sample size
Testing on low-traffic pages where significance is impossible to reach
Expecting to detect small differences with small samples

Frequently Asked Questions

Related Terms

Statistical Significance

The probability that observed differences between variants are real rather than random chance.