Christian Little

A few days ago I posted a big entry about setting up and displaying a complete multi-variant test on one of my client websites. This morning we flipped the switch and the test has started. So while Google’s Website Optimizer is busy flipping content variations, I have some information to show you so you’ll be prepared for your own testing in the future.

Scope of the Test

The first important thing you have to know about MV testing is that even though you can get 10,000+ variations of a single page with a test, that may be overkill. I actually fell victim to this until one of the staff at my client site reminded me.

Recall that the site in question gets about 40 sales per day on average. So from that figure alone, we can only expect about 30-40 variations to get a sale each day. So what’s the point of having 10,000+ variations if I don’t have the conversion volume to properly test it? All that would do is skew the data.

So after a few days of designing variations, I originally had 1000 variations for the page which just won’t work. That was overkill, so we cut it down to just 20 variations. That way each variation might get about 2-3 sales (which is better than before!).

As you can see, Google confirms exactly how many combinations you are going to have for the test before you launch it.

Now it also tells you how many page sections I created (6), but we didn’t end up using some of them. We had a page section for a McAfee ScanAlert box which is in the sidebar of the site. I think we’ll test this in the future, but for now I disabled it. The purpose of that box was to toggle is on/off just to see if it really did help.

Some companies that put seals on your website (most notably HackerSafe) make these big claims about websites increasing their conversion rates by 1% by having their seal on it. And they do have some very impressive documentation on it, but it’s always good to test something like that. However that ones just one of the more minor items that I wanted to test, so I took it out.

The main sections that are being tested are as follows:

1. Headline Text – the homepage has a big block of text inside an H1 tag. It’s about 8 words and it really stands out, so we’ve setup 5 variations of this block (which is the ‘Text Below Header’ section shown in the image above).
2. Sidebar Arrangement – the sidebar has 4 product packages showing at the top of it in a specific order. Naturally I wanted to see what order works best, so we’ve added a few variations to the order that these get displayed.
3. Peel-Away Ad – there is a flash peel-away ad in the top-left corner of the site. I’ve created a section to toggle this on/off.

That’s it for the content variations. It sounds kind of dull and boring, but minor changes like this can make a massive difference.

Ready and Launch It!

The content variations are all set, so I’ve flipped the switch and Google has started displaying the new content to our visitors:

Now ideally this test will run for 5-7 days, but I’m going to keep a very close eye on it and see how it goes. Google might identify a winner within 48 hours if the results are very strong around a specific combination.

Once you launch a test with Google, you can then go back to your Website Optimizer homepage to see the basic information about the test:

Additionally, you can access the two main reports right away (but there is no data to show since the test just started).

This is the conversion report. It shows you a is of combinations that are being run and will show how they perform against the control variation (the Original). As time goes by, you’ll see these fields and slider bars populate as data is collected for each variation.

This is the page section report. As you can see, it’s the same type of layout as the previous report, but it has a few extra fields – most notably the 0/5 box next to each section – that is the Relevance Rating of each section. The RR tells you how much impact each section has on the performance of the test – this is very useful as a strong RR is a good identifier of sections that should get further testing down the road.

For example, if I find the Peel-Away Ad section gets a RR of 0 or 1, I know that it makes very little difference so we’ll just leave it in. However if it had a RR of 4 or 5, we know that it makes a very big difference to our visitors and will continue testing it in future tests. The RR is a great way to filter out unimportant page sections for future testing.

That’s all for now folks, if you have any questions feel free to ask away

Related Blogs

• Original MV Test Post
• Multi-Variant Q&A with Bryan Eisenberg