Sample Size
The number of visitors or conversions needed to reach statistical significance.
The number of visitors or conversions needed to reach statistical significance.
Why It Matters
Testing with too few visitors produces unreliable results—you might implement a change that actually hurts conversion. Calculate required sample size before testing to know if a test is even viable for your traffic levels.
Practical Example
Scenario
A home goods brand wants to test a checkout redesign. Current conversion: 2.5%. They want to detect a 10% relative improvement (to 2.75%).
Calculation
Sample size calculator shows: 15,000 visitors per variant needed for 95% confidence, 80% powerResult
With 40,000 monthly checkout visitors, they can run this test in ~3-4 weeks. A smaller test detecting a 5% improvement would need 60,000 per variant—3+ months.
Pro Tips
- 1Calculate sample size before any test—use free calculators like Evan Miller's or Optimizely's
- 2Smaller expected effects require larger samples (detecting 2% lift needs 10x the sample of detecting 20% lift)
- 3Focus on high-traffic pages where you can reach significance faster
- 4Consider testing macro-conversions (purchases) vs micro-conversions (add-to-cart) based on volume
Common Mistakes to Avoid
Frequently asked questions
Put it to work
Related Terms
ONE FREE AUDIT
Run your free Shopify audit.
50+ factors. Real fixes. Same framework we run for $10M brands.
Free · No credit card required
You'll talk directly to the people doing the work.
7.7x blended return at scale — seven figures attributed, across five straight quarters · Modern Prairie · Apr 2025–Jun 2026 Read the Modern Prairie case study
