How to A/B Test Your Shopify Popups (and Which Metrics Matter)

Two popups can look almost the same and perform very differently. Across PopupBoost campaigns with at least 500 impressions, the bottom quarter gets 3.4% or less of viewers to engage, while the top 10% get 36.8% or more.

That gap is not luck. It comes from decisions you can test. This guide shows how to run those tests without fooling yourself.

Data note: Benchmarks come from aggregated, anonymous PopupBoost data, February to November 2026.

What A/B testing means for popups

An A/B test shows two versions of your popup to two random groups of visitors at the same time. Everything stays the same except one thing. After enough visitors, you compare the results.

The "same time" part matters. Comparing this month with last month mixes up your change with everything else that changed, such as traffic sources, season and promotions.

What to test first

Not every change is worth a test. Start with the ones that, in our data, are linked with the biggest differences.

1. Timing

Median engagement for popups that appear immediately was 12.0%, against 4.9% for a 4 to 6 second delay. If your popup waits, this is the first test to run. See popup timing on Shopify.

2. The prize

Free shipping prizes turned into orders 15.0% of the time, against 4.4% for percentage discounts. Test your current prize against free shipping, or a smaller discount against a bigger one. Read spin wheel prize ideas.

3. Form fields

Campaigns that asked for a name had a median engagement of 5.2%, against 8.3% without. Test removing a field before you test anything visual.

4. The game

Scratch cards and slot machines had higher median engagement than spin wheels in our data. Test the format with the same prizes. See scratch card vs spin wheel.

5. The headline

"Spin to win up to 25% off" vs "Get a gift on your first order." Copy tests are cheap and can move results.

6. Design

Colors, fonts and images come last. They matter, but usually less than the five tests above.

Pick the right metric

Each step of the funnel gives you a different metric:

For timing, game and copy tests, engagement or signup rate is usually fine. For prize tests, you must look at orders, because a bigger prize can raise engagement and still lower profit.

How many views you need

This is where most popup tests go wrong. A test that ends after 200 views per variant will often crown a winner that is just noise.

As a rough guide, to detect a change in engagement from 8% to 10% with standard confidence (95%) and power (80%), you need about 3,200 views per variant.

Orders need much more. The median PopupBoost campaign produces about 5.7 orders per 1,000 views. Detecting a 20% improvement in orders at that rate takes tens of thousands of views per variant.

That is why many stores test engagement and signups first, and only test for orders once they have enough traffic.

Run it for at least a week

Even with enough views, run every test for at least seven days. Shoppers behave differently on weekdays and weekends, and on paydays and quiet days. A test that runs Monday to Wednesday only tells you about Monday to Wednesday.

Also avoid testing across big events. A test that starts the week before Black Friday and ends the week after will mostly measure Black Friday.

Fixed split or bandit?

PopupBoost supports two kinds of experiments:

• Fixed split: traffic is divided evenly for the whole test. You get the cleanest comparison.
• Bandit: traffic gradually shifts to the variant that is doing better. You lose fewer visitors to a weak variant while the test runs.

Use a fixed split when you want a clear answer you can trust and reuse. Use a bandit when you have a few options and mainly want to earn more during the test.

Mistakes to avoid

• Changing several things at once. You will not know which change caused the result.
• Stopping as soon as one variant leads. Early leads flip all the time.
• Judging prizes by engagement. A bigger discount gets more spins and may lose money.
• Testing on tiny traffic. If your popup gets 500 views a month, test only big changes, such as timing.
• Not writing results down. Keep a simple log of each test, the dates, the numbers and the decision.

A simple testing plan for the next three months

1. Month 1: test timing (immediate vs your current delay).
2. Month 2: test the prize (your current prize vs free shipping with a minimum order).
3. Month 3: test the game (spin wheel vs scratch card) with the winning timing and prize.

Each test builds on the last winner. By the end, you will have a popup tuned to your own traffic, not to a general best practice.

Your next step

Pick one test from the list above, the one most likely to matter for your popup, and set it up today as a fixed split. Write down what result would make you switch, before you see the numbers.

For the full picture of what makes a popup work, start with our complete guide to Shopify popups.

Frequently asked questions

How long should I run a popup A/B test?

Run it until each variant has enough views to detect the difference you care about, and for at least one full week so weekday and weekend traffic are both included. Detecting a change from 8% to 10% engagement needs roughly 3,200 views per variant.

What should I A/B test first in my popup?

Start with the changes that move results most in our data: timing (how fast the popup appears), the prize, and the number of form fields. Colors and fonts usually matter much less.

Which metric should decide the winner?

Use the metric closest to revenue that has enough data. Engagement and signups fill up fast, so they are good for early tests. Orders and revenue per visitor are the real goal, but they need much more traffic to measure reliably.

What is a bandit test?

A bandit test shifts more traffic to the variant that is performing better while the test is still running. It loses less traffic to a weak variant, but it gives a less clean read than a fixed 50/50 split. Use a fixed split when you want a clear answer, and a bandit when you mostly want to earn more during the test.