A/B Test

A controlled experiment comparing two versions to determine which performs better.

1 min readLast updated Apr 2026

Reviewed by Golden Digital·Operator-reviewed ecommerce reference

Quick Reference

CategoryConversion Rate Optimization

Related Terms2

A controlled experiment comparing two versions to determine which performs better.

Why It Matters

A/B testing replaces opinions with data. Instead of guessing whether a red or green button converts better, you test both and let customer behavior decide. Even small wins (5-10% improvements) compound across thousands of visitors into significant revenue gains.

Practical Example

Scenario

A jewelry brand tests two product page layouts: A (current) vs B (larger images, simplified description). They split traffic 50/50 for 3 weeks.

Calculation

Variant A: 2.8% conversion. Variant B: 3.4% conversion. With 50,000 monthly visitors and $120 AOV.

Result

Variant B wins with 95% confidence. Implementing it adds 300 extra conversions/month = $36,000 annually from one test.

Pro Tips

1Test one element at a time to understand what caused the change
2Run tests until you reach statistical significance (95% confidence minimum)
3Calculate required sample size before testing—most tests need 1,000+ conversions per variant
4Document all tests and results to build institutional knowledge

Common Mistakes to Avoid

Ending tests too early before reaching statistical significance

Testing too many elements at once, making it impossible to know what worked

Not considering segment differences—a winner overall may lose for mobile users

Frequently asked questions

Until you reach statistical significance (usually 95% confidence) AND have run at least 1-2 full business cycles (to account for day-of-week variation). Minimum 2 weeks for most ecommerce sites.

Start with high-impact pages (product pages, cart, checkout) and high-visibility elements (headlines, CTAs, hero images). Test where you have the most traffic and the biggest conversion opportunities.

Put it to work

Related Terms

Multivariate Test

Testing multiple elements simultaneously to understand how different combinations interact.

Statistical Significance

The probability that observed differences between variants are real rather than random chance.

ONE FREE AUDIT

Run your free Shopify audit.

50+ factors. Real fixes. Same framework we run for $10M brands.

Run a Free Audit Tell us about your brand

Free · No credit card required

You'll talk directly to the people doing the work.

7.7x blended return at scale — seven figures attributed, across five straight quarters · Modern Prairie · Apr 2025–Jun 2026 Read the Modern Prairie case study