Split testing
Split testing is also called A/B testing and is also known as multivariant testing. Here you can read what it is and what you should use it for.
Imagine you have a webshop. Your conversion rate is 3% (i.e. three out of every 100 visitors to your webshop become customers). You'd like to raise it to 4%, because you'll earn 33% more on the same amount of traffic. But that's easier said than done.
To increase the conversion rate, you take a critical look at your shop and think "what's holding my users back from buying something?". You realize that many people worry about delivery time. So you want to make that clearer. So you write "Day to day delivery" right under the "add to cart" button.
And your conversion rate goes up. That's great.
But you don't really know if it's because you added the text "Day to day delivery". It could be due to many things - for example, that you're running a sale during the same period. So right now your conversion has increased and you're happy, but you don't know if it was your change that did it.
And that's exactly what you can use split testing for.
A split test will show your original Add to cart button to half of the users and the new Add to cart button to the other half. The tool will then also register which of them ends up buying something. This way, you will know whether your new addition had the desired effect or not. Let's take an example where you iterate it.
On January 1, your conversion rate is 3%. From January to March, you run split tests where you split test your original button with the new button. It turns out that the new button has a conversion rate of 3.3% against the new button's 3%. So, of course, you implement the new button on all your pages and you now have a conversion rate of 3.3% (a 10% increase in conversion).
You then observe that many people get stuck in your check out flow. So you start a split test where you cut down on the number of forms. The test runs from April to June and when it's over, you realize that those who saw the new check out flow have a conversion rate of 3.5%. Now you implement it and enjoy a 3.5% conversion rate.
And you can go on and on and on. Many large shops always have at least one split test running and slowly improve their conversion rate.
The numbers behind a split test
To be on the safe side, you need a tool to record the numbers for a split test. The tool should record:
- How many have seen version A and version B respectively
- How many who have seen version A have converted
- How many who have seen version B have converted
And once you know those numbers, the rest is math. Below you see an example.
| Version A | Version B | |
|---|---|---|
| Number of visitors | 2300 | 2370 |
| Number of conversions | 54 | 58 |
| Conversion rate | 2,34% | 2,44% |
As a starting point, you might say "Version B is best". But the improvement is so marginal that you would say the test is not valid. There is simply no statistical evidence to say that version B is better than version A. When the difference is so small, it could be due to chance. You would say that the test is inconclusive, and unfortunately this happens often.
When this happens, there's nothing to do but note down that the test was run and it had no noticeable impact. On to the next test.
When can you trust a split test?
Most people have a rule of thumb that says the confidence interval should be 95%. This means that you have a good 95% probability that the improved version will convert better than the other version.
Once you have the numbers you need, you can use this tool to calculate the confidence interval. That way you don't have to do the math yourself.
What can you split test?
Well, almost anything. Here are some of the obvious things to split test on your online store.
- The design (add to cart button (size, wording, color, placement), USPs, structure of the check out flow)
- Your Facebook ads (images, wording)
- The text of your Google Ads
- Newsletters (subject line, content, wording)
- Discount codes you send with orders (gives a 10% discount for orders more than 50,- off)
- Pop-ups / slide-ins - do you get more newsletter sign-ups by showing a puppy instead of a kitten?
The list is endless. There are plenty that can be split tested and constantly improve your online store.
Note, however, that split tests on emails are a bit special. Because it's rare to send the same newsletter multiple times. So, for example, if you have 50,000 subscribers to your newsletter, you can split test two subject lines and send to the first 20,000 recipients. You then find out that subject line B was the best, so you send it to the last 30,000 recipients of your newsletter. And then the party ends just like that, as you can't use the subject line again for the next newsletter. But you can use your experience from that one for the next time you write a newsletter.
Tools for split testing
Most advertising tools today have split testing built in. This is both Google Ads and Facebook's ad manager.
Most email marketing tools also have built-in split testing for both subject line and content.
The biggest challenge is split testing on your website, and Google Optimize is probably the most popular choice as it's free and fairly easy to set up.
Note - it does require traffic.
If you only run a small shop, such as a webshop for your sports club, there probably won't be enough visitors to your shop to run an effective split test. You have to go with best practice and your gut feeling.
We know online marketing in Shoporama
We've been working with online marketing ourselves for decades. As the only shop system in the country, we have spoken multiple times at conferences such as Marketingcamp, SEOday, Shopcamp, Digital Marketing, E-commerce Manager, Ecommerce Day, Web Analytics Wednesday and many more.