Sample Size
The number of visitors or conversions needed to reach statistical significance.
1 min readLast updated Apr 2026
Reviewed by Golden Digital·Operator-reviewed ecommerce reference
Quick Reference
CategoryConversion Rate Optimization
Related Terms1
The number of visitors or conversions needed to reach statistical significance.
Why It Matters
Testing with too few visitors produces unreliable results—you might implement a change that actually hurts conversion. Calculate required sample size before testing to know if a test is even viable for your traffic levels.
Practical Example
Scenario
A home goods brand wants to test a checkout redesign. Current conversion: 2.5%. They want to detect a 10% relative improvement (to 2.75%).
Calculation
Sample size calculator shows: 15,000 visitors per variant needed for 95% confidence, 80% powerResult
With 40,000 monthly checkout visitors, they can run this test in ~3-4 weeks. A smaller test detecting a 5% improvement would need 60,000 per variant—3+ months.
Pro Tips
- 1Calculate sample size before any test—use free calculators like Evan Miller's or Optimizely's
- 2Smaller expected effects require larger samples (detecting 2% lift needs 10x the sample of detecting 20% lift)
- 3Focus on high-traffic pages where you can reach significance faster
- 4Consider testing macro-conversions (purchases) vs micro-conversions (add-to-cart) based on volume
Common Mistakes to Avoid
Starting tests without calculating required sample size
Testing on low-traffic pages where significance is impossible to reach
Expecting to detect small differences with small samples