Introduction to A/B & Multivariate Testing in Web Design

Anyone who is familiar with increasing website conversions has probably heard about A/B testing. It is a way for webmasters to display 2 variants of the same page out to their audience at random, and then gauge results based on user performance. A/B testing(or split testing) is very popular on sites where users are more interactive on the page. Think about projects like social networks, SaaS products, e-commerce shops, business portfolios, etc.

split testing ab dribbble shot philip clark

There is also a higher stage of this process called multivariate testing. Here you can include more than just 2 options which may range out to A/B/C/D+ testing. The concepts are all similar but you’ll need to run tests for longer in order to acquire accurate results. In this guide I want to introduce the idea of split testing and how webmasters or project directors can get started.

What to be Testing?

This is a common first question and it’s completely reasonable to ask. You want to think about areas on your webpage, newsletter, or other digital interface which could be improved for an easier user performance. Blog headlines, sidebar advertisements, main navigation links, anything which should be commonly accessed by users is fair game.

The goal is to see which of your 2 designs performs best in a general 50/50 split of your traffic. You want to determine the average percentage of users who click your headlines more frequently, or click on the quicker, or something like this. It’s all about “fixing” your website’s first impression to get users more interactive on the areas you want them to be interacting with.

Online Tools

I would recommend a list of tools, but many of them do charge money for a full account. SaaS products have become more relevant and it is often easier for webmasters to pay money to a 3rd party service and have them manage the backend.

If you want a free product then definitely try out Google Website Optimizer. Their old URL has gone down and the service is now combined into Google Analytics. But it is completely free to use and directly ties into your Analytics traffic results. This is the best way for a new tester to dive right into the process.

visual website optimizer vwo homepage layout screenshot

But when you know you want something more detailed, I would highly recommend Visual Website Optimizer. You can signup for a free plan which has some pretty relaxed limits. It does include a majority of the paid plan features – however you are limited to testing only 1,000 visitors per month. For high-traffic websites it’ll take a lot longer to obtain useful results. But on low-traffic websites this is a great way to start manipulating your interface and seeing how it affects conversion rates.

Newsletter Designs

Not every website needs to or should run a newsletter. But for larger communities it is the perfect way to keep people visiting over time. You can include new items for sale, free downloads, or recent blog posts. And it is a great way to check out the response from visitors onto your internal landing page(s).

A/B testing for newsletter designs is much more common than you might think. It is somewhat easier than testing a website, and it can be handled by many e-mail delivery services. Both Campaign Monitor and MailChimp offer split tests as a part of their system. You can build two distinct email layouts and have them divided into a rough 50/50 split out to your subscribers.

email newsletter ab testing split multivariate mailchimp diagram

Obviously your goal for testing here is the same as anywhere else. How can you redesign buttons, links, headers, and content to engage visitors into clicking through to your website? Once subscribers can flesh out your newsletter they will be more likely to actually read what it says. Maybe one of your links or headlines gets somebody curious and they go through your entire landing page.

Web copy is just as powerful as small design changes. Many case studies online talk about this minor difference which can have a tremendous impact on your conversion rates. Don’t try to assume minor updates are silly and will have no effect, because you may be surprised when you see the results!

Testing in Phases

I want to add that you can’t give up too easily on this stuff. Waiting around for 5,000 visitors may not be enough of an audience to completely gauge how you changes are working. Don’t be too steadfast to move onto your next idea without allowing enough time to pass.

The best way to manage your A/B testing is in phases of updates. Either look for small wins through single-element updates, or try updating a lot on the page and see which version of the layout works best. These two scenarios are both helpful and they can both be run using an A and B option. But what you are testing for will play a huge role in the process.

android ux ui testing mobile app screenshot dribbble

If you feel more comfortable doing minor changes between design iterations this is another solution. Just be sure that you leave time for enough visitors to check out the page and give you some helpful feedback. Writing down a full list of ideas can be a great start towards figuring out what you hope to test, and how these tests may improve the percentage of user interaction.

Related Websites

There are some really cool blogs online which discuss the ideas of A/B testing. Some include tips while others focus on case studies and real impactful results. Check out some of these links if you want to learn more.


There are plenty of similar online articles which delve into the A/B testing process. I would greatly recommend looking for case study articles in Google. These will provide context for what other webmasters have tested, and what the results were. It may actually save you time in the long run when brainstorming ideas that you should be checking out on your website. A/B split testing most likely isn’t needed for every website, but when understood properly it can provide value into any creative project.

Jake is a researcher and writer on many design & digital art websites. He frequently writes on topics including UX design, content marketing, and project management. You can find more work on his portfolio and follow his latest tweets @jakerocheleau.

Leave a Reply

Your email address will not be published. Required fields are marked *

Host Up to 100 Websites with the Media Temple Grid Plan