Christian Little

Multi-Variant Analysis (or MV for short) is a very advanced method for improving the performance of your website. It requires advanced software, as it is pretty much impossible to do it on your own unless you are an android like Mr. Data on Star Trek.

Note: There are several accepted spelling formats including: “Multi-Variant”, “Multi-Variate”, “Multivariant”, and “Multivariate”. Spelling of this testing methodology depends on where you are from (North America, the UK, France, or other parts of the world use their own variation on the name).

What’s involved is identifying sections of a page, and then coming up with multiple variations of content for those specific sections. Some examples include:

Header Image – Most website have a header image, you can try a few different header images during the test to see what your visitors like.

Body Test – Usually tested 1 paragraph or heading at at time, you can change the text, the CSS, or anything else about it. Maybe even test if a plain text heading works better than an image heading.

Using just those 2 sections, lets say for example that you have come up with 3 variations for your header image and 5 variations for the first text header in your page content. This gives you the following information about your test:

Header Image: Original + 3 Variations = 4 Variations
Text Header: Original + 5 Variations = 6 Variations

Possible Combinations = 1 x 4 x 6 = 24 Combinations

Now the purpose of the testing platform is to randomly display one of those combinations to different visitors, and make sure that every combination gets displayed equally. So if you had 5,000 page views the testing platform would display each combination 5000 / 24 = 208 times during the test.

Once the test has completed, the platform will give you a detailed report showing you how every single combination performed with respect to your goal. So you might find that your visitors prefer one of the new headers and your original text header, so you put that combination onto your website and watch as your visitors convert better.

It’s pretty simple to understand, but the math and programming logic behind it is extremely complicated, which is why there are only a few reliable testing platforms on the market today.

This Sounds Expensive

It can get pricey when you start looking at the higher end testing platforms (which can cost tens of thousands of dollars per year!). But thankfully, the good folks over at Google came up with a free testing platform called Website Optimizer. It’s not as advanced as the paid ones, but it works well so it’s a good starting point.

The reason for using the expensive commercial versions of this software is when you want more features. Website Optimizer is a great basic version, but it has several limits, such as:

• You can only test 8 sections on a single page. The paid versions usually don’t have a limit.
• You can’t run mutli-page MV tests. Some of the more expensive paid versions (such as Omniture’s Test & Target) can do this.
• Conversion are only tracked by the number of people that make it to your conversion page, whereas the paid platforms take it a step further by analyzing average order size and other ecommerce variables to make the test way more reliable.

UPDATE: I was contacted by a Google representative shortly after making this post and they pointed out that the Website Optimizer now does support multi-page MV testing. After reviewing the documentation this morning, it looks like it will work but it’s basically a hack of their code for a single page test. Still, it is promising.

How to do it: Google Multi-Page MV Hack

These are just a few of the differences, I can list off dozens more but that’s not the point of this posting.

This experiment will be run using Google’s Website Optimizer so you can see how it works.

Something Important to Consider

The most important thing to note when it comes to MV testing is that you need to get at least 10,000 page views per week. Otherwise you need to use a very small experiment size (limit to 2-3 sections with only a few variations). The reason for this is pretty simple:

• With 8 sections and 5 combinations of each, you can easily get over 10,000 combinations to test.
• With 3 sections and 3 combinations of each, you will get around 150 combinations.

Basically this comes down to the scope of the project. If you have thousands upon thousands of people coming to your sites, then you can do a broader experiment. If you try to do the same experiment on a site with only hundreds of visitors, the data will be completely skewed as you won’t get that many people hitting the conversion page in the first place.

The Test Site

I have displayed two of the key metrics for this specific test for the test site in question. The statistics were collected by Site Catalyst (my favorite analytics platform), and we have verified with the accounting department about the accuracy of the revenue tracking when it was finally put into effect on September 9.

NOTE: Due to legal implications, I cannot disclose the URL of the test website. Please do not ask.

As you can see from the the chart above, the website averages about 7,000 pageviews per day. That’s 49,000 per week, which is more than the minimum needed to conduct a good MV test.

The revenue chart above shows the website averages about $8,500 in revenue every day, or $59,500 per week.

NOTE: You don’t need to have a website as successful as this one to run MV testing. As mentioned above, you really only need about 10,000 page views per week to run one.

The Experiment

Based on the above information, and some other details that I pull out of Site Catalyst, we’ve determined the website converts 1.60% of visitors into sales. But the company average is 2.00-3.00% conversion rate on most of their websites, so we want to see if we can bring this up to that level.

Here are the important numbers:

The above chart is just the average data. I’ll post a more precise chart for this data later on during the testing period. But you can see that if the test is able to raise the conversion rate to 2.00% they will see an increase in daily revenue by $2,000 on average, which works out to a 25% revenue gain per month!

Keep checking back, the next post on this experiment will cover how I divided up the page sections and setup the test in Google’s Website Optimizer. After that, the test will begin!

Related Websites

• Google’s Website Optimizer (free A/B and MV testing platform)
• Compucall Web Marketing demonstrating a MV test with Google Website Optimizer
• MV Testing Panel at SES San Jose 2008