What is A/B testing?

A/B testing or split testing is comparing two versions of a web page to see which one performs better. It’s a useful method for figuring out the best promotional and marketing strategies for your business, and it could be used to test everything from search ad keywords, email headlines, ad copy, to website design. For example, splitting users between two CTAs, one of which says “Learn More” and the other “Start your free trial” and let them run for a while then measure each of their conversion rates.

Why should you do A/B testing?

A/B testing could be valuable for companies because they are low in cost but high in reward. It helps you uncover insights, whether your test wins or loses. The value is very transferable, some of the advantages for an optimized website are increased conversion therefore revenue and satisfied clients. A/B testing allows you to give your site’s visitors the ultimate decision on whether one design outperforms the next.

How to do A/B testing?

In an A/B test, you’ll show 50% of visitors version A (Control) and show the rest of the 50% version B (Variate). During the test, version A would remain the same in order to compare the tweaks you made on version B. If the variant yielded a higher conversion rate, it becomes the new control and you must design a new variant.

( Photo source: Optimizely )

You can almost A/B test everything on your e-Commerce website, here are some ideas for you to think about:

• Adding a sales section or button which leads shoppers to products that are on sale.
• Emphasize best selling products and deals on your homepage, and help customers to make the decision faster.
• Change the color of your CTA button
• Offer more payment or shipping options.
• Add customer reviews on your product page.

When performing an A/B testing, there are some rules to keep in mind:

1. Allow your test to run for at least 7 days to get enough data and sample size

Make sure that you let your test run long enough in order to obtain a substantial sample size. Otherwise, It’ll be hard to tell whether there was a statistically significant difference between the two variations.

2. Change just one variable at a time

When you’re planning A/B testing, make sure you have a clear goal such as conversion rate, website traffic, or sign up rate, and more. For example, if you want more visitors to click the “BUY’ button, to improve the chances of this, try not to tweak other designs in the page. If you change too many elements at once, it gets hard to evaluate what variation results in a higher click rate.

Testing more than one thing at a time is a multivariate test, and is more complicated to run. We’ll cover this in another article.

3. Testing assumptions

The best thing about A/B testing is that sometimes the results would come up unexpectedly, and help eliminate false assumptions. The lesson is to always be testing, so you can test assumptions and make sure each change improves conversion.

4. Your test should run until there’s at least a 95% chance to beat the control

Sometimes it’s tempting to end a test earlier, but this is the only way to know for sure that the conversion rate improvement is statistically relevant. You also should know that the smaller the improvement, the longer the test will need to run in order to reach statistical significance. A test with a 5% improvement will need to run longer than a test with a 45% improvement.

Now that we have that out of the way, let’s talk about how some tools could help you with A/B testing:

SHOPLINE: Our website builder allows users to build both single page or multi-page websites. Simply drag and drop the contents or photos to build your store, without any coding experience. Users can build various pages to perform A/B testing and inset UTM parameters to track results on Google Analytics. Check out how to build single page with SHOPLINE here.

Google Content Experiments: With Google’s content experiments, users can test which version of a landing page results in the greatest improvement in conversions. It works a little different than standard A/B testing, while you’re not just testing two versions of a page as in A/B testing, instead, you are testing up to 10 full versions of a single page, each delivered to users from a separate URL.

Optimizely: A landing experimentation platform that allows anyone to learn more about their audience and experiment with design and content. Easy to get minor tests up and running, even without technical skills.

MailChimp: An email marketing tool, users can send out two identical emails with one variable that is different. The variable could be time sent, the title, copy, photos or more, and see how small changes can have an impact on your results.

Conclusion

We’ve explored some of the most important overarching principles for A/B testing. Just remember the “Big Three” factors of A/B testing and keep them intact from start to finish in your testing process:

• Form the right hypothesis — no wild guesses or gut feelings.
• Keep going until you reach 95-99% statistical significance.
• Make sure your sample size is large enough (at least 1,000 conversions).
• Don’t stop running your test too soon. Aim for 1-2 weeks.

If I had to sum up my best advice in four words based on my real-life experience with A/B testing, I would say this: be precise and be patient.