How to create and run a split test in Shoporama
Step-by-step guide to create and run a split test (A/B test) in Shoporama. You will learn how to clone a page setup, stylesheet, or split test entire themes, create the test, read the statistics and let the system find the winner automatically.
With split testing, you can compare two or more variations of the same page, stylesheet, or entire theme to see which one actually performs best. Here’s a step-by-step guide to setting up and running your first split test. Read the feature description if you want to understand how the system works behind the scenes.
Prerequisite: You must have two variants
A split test compares two or more variations of the same element. Before you can create the test, the variations must already exist. There are three types:
- Page layouts — created in the Page Designer under Design → Page Designer. The Page Designer is only available in themes that support it, such as Alaska2, California, Delaware, Montana, and Washington
- Stylesheets (CSS variants) — created under Design → Stylesheet
- Entire themes — exist as folders under the store’s private themes. Requires a Pro subscription and Smarty 4. You do not create the themes themselves via the Split Test page—they must already be present in the store (uploaded via FTP, copied from an existing theme, or built by a developer).
Step 1: Clone the page or stylesheet you want to test
Go to Design → Page Designer (or Stylesheet if you want to test CSS). Find the page you want to test in the list, and click the copy icon (the two sheets of paper) on the right side of the row.
You’ll now get a copy that starts offline with the name “(original name) (copy).” The clone automatically has a lower priority than the original, so if you put it online without testing it, the original will still be displayed.
Step 2: Customize the copy
Click the edit icon (pencil) next to the copy to open it. Make the changes you want to test:
- Change the hero image
- Edit the headline or text
- Add or remove sections
- Reorder the sections
- Edit CSS (if using a stylesheet variant)
Important: Test only ONE thing at a time. If you change five things and B wins, you won’t know what made the difference. Start with, for example, just the headline, or just the button color.
Step 3: Publish the copy
When you’re done making changes, go back to the overview and click the toggle icon next to the copy to put it online. It turns green when it’s active.
You now have two online versions of the same page. Without a split test, the one with the highest priority would automatically win and be shown to everyone. This is where the test comes in.
Theme test: a slightly different workflow
If you want to A/B test entire themes against each other, Steps 1–3 do not apply. Instead, you must already have two private themes on the store—for example, MyTheme and MyTheme-v2. On the create page, select “Theme” from the dropdown menu, and on the edit page, add the two (or more) theme folders you want to test. The rest of the process (weighting, statistics, winner) is identical to that for page layouts and stylesheets.
The “Theme” option only appears if the store has a Pro subscription, runs Smarty 4, and has at least two private themes. The Pro requirement is because private themes are a Pro feature in and of themselves. The Smarty 4 requirement is technical: only Smarty 4 can render two different themes side-by-side on the same store without their template cache going haywire.
If you’ve enabled “Use the winner as the new default when the test ends” and the test ends with a valid winner, the winning theme is automatically set as the store’s new default theme (equivalent to selecting it yourself under Design → Theme). Otherwise, the theme won’t change, and you’ll have to choose it yourself. The other theme folders remain untouched. They’ll still be there if you want to test them again later.
Step 4: Create the split test
Go to Design → Split Test and click New Split Test.

- Give the test a name so you can find it again later. For example, “Homepage Hero A vs. B” or “Button Color: Red vs. Green.”
- Select what you want to test from the dropdown menu. You’ll only see the target types for which there are at least two variants.
- Click “Create Split Test.” You’ll be taken to the test’s edit page.
Step 5: Add variants to the test
On the test’s edit page, you’ll see a “Variants” section. Use the dropdown menu to select a variant and specify a weight (typically 50). Click “Add.” Repeat for the second variant.
The weight determines what percentage of visitors will see each variant:
- 50/50 — recommended. The fastest way to reach a conclusion because the data is distributed evenly.
- 90/10 — use this if you’re cautious and only want to expose a new variant to a small portion of traffic. It takes longer to gather enough data.
Step 6: Configure settings and start
On the right side of the edit page, you can set:
- Start and end dates — optional. Leave the end date blank to let the test run until you stop it yourself.
- Stop automatically when there is a clear winner — recommended. The test stops when there is a ≥95% probability that a variant is the best and the minimum requirements are met (at least 1,000 visitors per variant, 100 orders in total, and 14 days since the test started).
- Use the winner as the new default when the test ends — recommended. When the test ends with a valid winner, the winning variant is put online with high priority and the losing variants are taken offline. You can always roll back manually.
Click “Start the Test ” when you’re ready. The button is disabled until you’ve added at least 2 variants.
Step 7: Wait and keep an eye on the statistics
While the test is running, you can return to the test’s edit page at any time to view the statistics. You’ll see:
- Visitors per variant (unique visitors who have viewed the variant)
- Orders and revenue
- Conversion rate as a percentage
- Revenue per visitor (RPV) —the most important KPI in e-commerce
- Probability of winning, shown as a colored bar. The color corresponds only to the percentage: gray below 70%, blue at 70% or higher, and green at 95% or higher. A green bar does not in itself mean that a winner has been selected, and the leading variant may well have a gray bar.
The leading variant is highlighted with a light green background and is labeled “In the Lead.” When a winner is declared with a 95%+ probability and the minimum requirements are met, it is labeled “Winner” with a trophy, and a green box appears at the top reading “Winner found:”.
Please be patient. The statistics are updated when visitors click on one of the variants. For a small store, it may take 4–8 weeks before there is enough data to draw a conclusion.
View results by device (mobile, tablet, and computer)
The same variant can easily win on a computer and lose on a mobile device. People shop differently depending on the screen: a new hero might look great on a large screen but push the buy button way down on a phone. That’s why you can view the numbers broken down by device in the statistics.
At the top of the statistics box, you’ll find the “Device” menu. It’s located next to the “From,” “To,” and “Days” fields, which you can read about in the next section. “Device” has four options:
- All devices — the overall picture (default).
- Computer — only visitors on desktops and laptops.
- Mobile — visitors on phones only.
- Tablet — visitors on tablets only.
When you select a device, the statistics update immediately, and the entire table is filtered so that you see only visitors, orders, revenue, conversion, and win probability for that device. The number in parentheses next to Computer, Mobile, and Tablet shows how many visitors were recorded on that device during the selected time period and on the selected days, for all variants combined. The rule of thumb is 1,000 visitors per variant, so you should divide that number among the variants before determining whether there is enough data.
Good to know: A device-specific view does not determine a winner, and an automatic winner is always selected based on the total numbers. This makes sense because your store can only display one version of a page at a time—you can’t have one homepage on mobile and another on desktop based on a split test. Use the device breakdown to understand the results, not to choose two different winners.
The device is determined based on the visitor’s browser, just like in the major A/B testing tools. The breakdown counts visits from the time the feature was activated on your store—older visits from before that are only included under “All devices.”
View a specific time period, weekdays only, or weekends only
The total doesn’t always tell the whole story. Perhaps the variants perform similarly on weekends but differently on weekdays. Perhaps a newsletter or campaign ran for a few days and attracted different customers than usual. Or maybe you’ve restarted the test and want to see how it’s been going since then. That’s why you can filter the statistics by a specific time period or by specific days of the week.
Next to “Unit,” you’ll find three fields as soon as the test has data:
- From and To: the period you want to view. By default, these are set to the first and last days of the test with data, so you see the entire test.
- Days: Choose between All Days, Weekdays Only (Mon–Fri), and Weekends Only (Sat–Sun).
If you select an option in “Unit” or “Days,” the statistics update immediately. If you’ve changed the dates, click “Show.” You can combine the fields, e.g., “Mobile,” “October 1–15,” and “Weekends Only.” The entire table reflects your selection, including the probability of winning and which variant is in the lead. Click Reset to view the entire test again. The filter is not saved, so if you leave the page or click Save, you’ll see the entire test again.
Both the From and To dates are included, and the entire day counts. If you select October 1–15, you’ll see everything from October 1 at 12:00 a.m. to October 15 at 11:59 p.m., according to Danish time. Weekdays are Monday through Friday, even if one of those days is a holiday. You can only view one continuous period at a time.

Good to know: A visitor is counted on the day they first saw the variant, and an order is counted on the day it was placed. Therefore, the conversion rate over a short period may appear higher or lower than it actually is. Compare the variants with each other within the same filter, rather than comparing a specific period to the entire test. The longer the period, the less meaningful the results are. Viewing data for a specific period or certain days of the week never determines a winner. “Automatic Stop” and “Use the Winner as the New Default ” always use the test’s overall figures.
Step 8: When the test ends
If you’ve enabled “Use the winner as the new default” and the test has a valid winner, this happens automatically: the winning variant is activated as the new default, and the losing variants are taken offline. You’ll receive a notification in the admin panel (the clock at the top of the page) explaining what happened.
If you haven’t enabled “Use the winner as the new default,” you can click into the test yourself, see which variant won, and manually update your store by going to Page Designer/Stylesheet and setting the winning variant to online (and, if necessary, the losing variants to offline).
The statistics are retained when the test stops. You can always go back and view paused and completed tests and their results, as long as you don’t delete them.
How to Stop a Test Manually
If you want to stop a test early, you can do so in two ways:
- From the split-test list: Click the toggle icon (green = running) to stop it.
- From the test’s edit page: Click the “Stop Test” button on the right-hand side.
Collected data is retained as history. If “Use the winner as the new default ” is enabled and there is a valid winner with enough data, the winner is set as the default. Otherwise, the test simply stops without any changes.
How to restart a test
A stopped test can be restarted by clicking the toggle icon in the split test list or by clicking “Start Test” on the edit page. Only tests with the status “Draft” or “Stopped” can be restarted. A completed test that was automatically stopped with a winner does not have a restart button.
Nothing is reset when you restart. The data from before the pause is retained and combined with the new data, and the 14-day period is still counted from the test’s initial start. If the test has been paused for less than 30 days, visitors who have already been counted are not counted again. If you want to see how the variants have performed since the restart, set the “From” date in the statistics to the restart date. Keep in mind that visitors who aren’t counted again can still place new orders, which are included in the count. The conversion rate after the restart may therefore appear too high, so compare the variants with each other rather than with the previous numbers. The winner is still determined based on all the numbers.
How to delete a test
Click the trash can icon next to the test in the list. A confirmation dialog will appear. Once the test is deleted, it cannot be resumed, and neither the test nor its statistics will be visible in the admin panel. Shoporama stores which variant the orders came from, but this information is not displayed on the orders or anywhere else. If you want to keep the results, stop the test instead of deleting it.
Time-Based Control Without Split Testing
You don’t need to run a split test to use the variant system. You can also simply create time-controlled pages:
- Clone an existing page (e.g., the homepage) and customize it to have a Black Friday look.
- Open the clone and go to Settings.
- Leave the “Online” checkbox unchecked.
- Set “Display from” to the start date (e.g., 11/27/2026) and “Display through” to the end date (e.g., 11/30/2026).
- Save.
The clone will automatically appear during the specified time frame (from midnight on the start date to 11:59 PM on the end date) and disappear again once the period is over.
Cookies and GDPR
When a split test is running, Shoporama may set a cookie called pb_v on your visitors’ browsers so they see the same variant on their next visit. The cookie is HttpOnly, SameSite=Lax, and expires after 30 days. It contains only a random visit ID and the variant the visitor is seeing. It is not shared with third parties.
Shoporama treats pb_v as a statistics cookie. If you use Shoporama’s cookie app, Cookie Consent, the cookie is set only if the visitor has consented to statistics. Otherwise, the variant is stored in the session instead, without pb_v. If you use a different cookie banner—or no banner at all—Shoporama cannot see the visitor’s choice, and the cookie will be set regardless of the visitor’s selection. The exception is the built-in cookie banner in the Alaska and Alaska2 themes: if the visitor selects “No” there, the cookie is not set. We recommend that you list pb_v in your cookie policy as a statistics cookie.
Frequently Asked Questions
Can I change the weights while the test is running?
No. The weight fields are locked while the test is active. If you want to change the weights, you must stop the test, adjust the weights, and restart it. Note that changing weights in the middle of a test can skew the results.
Can I add a new variant to a running test?
No. You can only add variants when the test has been stopped or is in draft status. This is to ensure data quality—adding a new variant in the middle of a test would result in a skewed sample size.
What happens if I delete a variant included in the test?
If you delete a page or stylesheet included in the test, the variant is automatically removed from the test, and its numbers disappear from the test statistics. The numbers for the remaining variants are retained. If a running test falls below 2 variants, it stops, and no winner is declared. If you want to see the variant in the results, stop the test and take the variant offline instead of deleting it. A theme you delete is not automatically removed from a theme test, so stop the test and remove the variant from the test before deleting the theme.
Where can I find historical tests?
All tests (drafts, paused, completed) appear in the list under Design → Split Test. The Status column shows whether the test is running, is a draft, has been paused, or has been completed with a winner.
Can I A/B test two different themes against each other?
Yes. If you have at least 2 private themes on your store + a Pro subscription + Smarty 4, “Theme” will appear as a third target type in the dropdown when you create a new test. You select two (or more) theme folders and run the test just like any other split test. If you’ve enabled “Use the winner as the new default when the test ends,” the winning theme will automatically become the store’s new default theme when the test ends with a valid winner. Otherwise, the theme won’t change.
What should I do if the “Theme” option isn’t there?
There are three reasons why it might not appear: (1) The store does not have a Pro subscription—upgrade under Company → Webstores; (2) the store is running Smarty 2—switch to Smarty 4 in the “Smarty version” field under Webstore → General (if it says “Controlled by the theme,” the theme determines the version). Older themes may not work with Smarty 4, so check the store afterward; (3) you have fewer than 2 private themes—upload or copy an additional theme first.
How do I know if a variant is online?
In the Page Designer list, the status column shows whether the variant is Active Now, Standby, Offline, Scheduled from X, or Expired. If the variant is part of an ongoing split test, it will say “In split test ” instead. The Stylesheet list shows the same statuses, except for “Standby” and “In split test.” The toggle icon is green when the variant is online.
Can the AI assistant create split tests for me?
No. The AI is intentionally unable to control the split test itself (creating the test, selecting variants, and starting or stopping it). These are decisions you must make yourself to ensure no one accidentally starts a test. However, the AI can create and schedule page setups when you describe them in your own words, for example , “Create a Black Friday homepage that automatically goes live on November 27 and is taken down after November 30.”
Will I receive a warning if traffic is unevenly distributed among the variants?
Yes. Shoporama checks the distribution once a day. If traffic is clearly unevenly distributed (e.g., 70/30 instead of 50/50), you’ll receive a warning in the admin panel. This is typically due to a cache configuration or bot activity. We send no more than one warning per week per test so you aren’t spammed.
What happens to my online store’s cache?
The cache is automatically disabled for the entire store—that is, all pages and all visitors—as long as at least one test is running. This ensures that each user sees the correct variant. Pages without the cache load more slowly, so don’t let a test run longer than necessary. Once a test is no longer running, the cache is re-enabled for new visitors. However, users who participated in the test will continue to have a non-cached experience for up to 30 days (the cookie’s lifespan), so we don’t serve them a cached version from a different variant.
Can I see if a variant performs better on mobile than on desktop?
Yes. Use the Device menu at the top of the statistics and select Mobile, Computer, or Tablet. You’ll then see visitors, orders, revenue, and conversions for that device only and can compare how the variants perform across screens.
Why can’t I choose a winner per device?
Because your store only displays one version of a page at a time to everyone. An automatic winner is therefore always selected based on the overall numbers. The device breakdown is there to help you understand the results—for example, if a variant that loses overall actually wins big on mobile—so you can make an informed decision.
Do I need Google Analytics to view mobile and desktop metrics?
No. The breakdown by mobile, tablet, and computer is built into the split test statistics. You don’t need to set up additional tracking in Google Analytics to see this.
A customer views the homepage on Friday and makes a purchase on Saturday. Does that count as a weekday or a weekend?
Both. The customer counts as a visitor on Friday because that was the day they first saw the variant, and the order counts on Saturday, when it was placed. So, under “Weekdays Only,” you’ll see the visitor, and under “Weekends Only,” you’ll see the order. This applies to all variants in the same way, so compare the variants with each other using the same filter.
Can the filter determine a winner or stop the test?
No. The probability of winning and the “Ahead” label are recalculated based on the numbers you’ve filtered, even if the bar turns green. But the filter is just a preview. It doesn’t declare a winner, stop the test, or change what visitors see.
Can I trust the numbers if there are only a few visitors left after applying the filter?
Be careful. Shoporama doesn’t display a warning when there are few visitors in the view you’ve selected. Use the same rule of thumb as when we declare a winner: at least 1,000 visitors per variant and 100 orders in total. Check the Visitors and Orders columns before drawing any conclusions.
Can I view the test data excluding a single week, such as a sale week?
No, not in a single view. You can only select one continuous period at a time using the “From” and “To” fields. Instead, view the period before and the period after the sale week separately. If you only want to see the sale week itself, set “From” and “To” to that week.
Where can I find more information about how this works?
Read the feature description of split testing to understand the statistics, GDPR compliance, and how we determine the winner behind the scenes.
Related articles
Split Testing of Theme Files (Legacy Feature for Smarty 2)
The old split test for individual theme files is only available for stores running Smarty 2. For new tests, use "Split Test" under "Design."
Set up a split test in Shoporama
A brief guide to setting up a split test (A/B test) under Design → Split Test and tracking the results.
Page designer
Lær hvordan du bruger Sidedesigner i Shoporama til at opbygge sidelayouts med blokke som hero-sektioner, produktlister, bannere og meget mere, helt...
Conversion rate - understand and improve your stats
Understand what your conversion rate means and how to use the statistics page in Shoporama to track your progress over time.
Related features
Test your front page against itself - split testing is live in Shoporama
Compare two versions of your front page, category page or stylesheet and see which one actually sells better. Bayesian statistics, automatic winner...
Split test
A split test compares two versions to find the best one. Learn what you can test, how to do it correctly and what pitfalls to avoid.
A/B Testing — Find Out What Works Best for Your Online Store
A/B test home pages, category pages, button colors, stylesheets, and entire themes against each other. Statistical winner, time tracking, and full...
Build your homepage in the page builder
The Shoporama homepage is a category. Build it using sections in the page designer, display the right products, and make sure the layout doesn't...