Christian Little

Hey look I’ve posted something new, hell must have frozen over! Nah, I’ve just been super busy (writing a thesis paper takes a lot of work, throw in working full time, family issues, personal health, dealing with the last of my MBA courses, and trying to find 5 minutes to relax and things just got a little out of hand). One of these days I will be back to my old posting frequency, but until then you’ll just have to make due with what I give you.

Image: Split Testing Process. Very cool process overview of how split testing works. Image courtesy of MindValleyLabs (link at the end of this posting).

By now you should have a basic understanding of what split testing is, if not check out some of my previous posts on the topic for some tests I’ve been running:

• Adventures with Google Website Optimizer Part 1 (aka Getting Ready to Launch)
• Adventures with Google Website Optimizer Part 2 (aka Test Launch)
• Adventures with Google Website Optimizer Part 3 (aka The Drama)
• Vertster Testing Platform

Onto the Metrics Already!

Enough with what is Split Testing, onto how to measure a split test. There are 9 key factors (at least in my opinion, the metrics vary from marketer to marketer). Here they are in a nutshell:

1. Conversions – The number of conversions (usually orders or signups) received on each variable.
2. Revenue (optional) – This revenue generated on each variable (not applicable on all tests as some sites are just trying to generate leads, not revenue.
3. Unique Visitors – You need to know exactly how many people saw each variable.
4. Revenue per Visitor (RPV) (optional) – How much money you made per-visitor (not applicable on all tests, see Revenue above for explanation).
5. Average Order Value (Avg $) (optional) – How much money you made per conversion (not applicable on all tests, see Revenue above for explanation).
6. Conversion Rate – A percentage showing the ratio of conversions to visitors.
7. Projected Conversions – This is a derived statistic in which you project your average traffic against the conversion rate to determine how many conversions you are likely to get over a future time period.
8. Projected Revenue – A derived statistic in which you project your expected revenue based on you average $ per conversion multiplied by your projected conversions.
9. Lift – The projected gain based on all other metrics.

Building a Split Test Summary Report

Using the above 9 metrics, I’m going to show you a sample report for a split test I recently ran. Here’s the report:

Sample Split Test Report

Yeah I know that’s really tiny image and hard to read, so I’m going to break it up into 3 parts and go over each part in detail.

Split Test Report Part 1

This first part is pretty straight forward. It shows the 3 variables in the test (1 control, and 2 test variables called v6 and v7). It also show the total orders, revenue, and visitors tracked on each version of the test.

At bare minimum, every split testing platform should automatically track this information in some form. If you can’t track these three simple metrics in your testing platform, then you’ve been ripped off and should find a replacement immediately. The remaining 6 metrics are derived from these 3 key metrics.

Split Testing Report

This next portion of the report shows the basic derivations you can make from the data that gets collected. It’s very simple to do the math here:

RPV (Revenue per Visitor) is simply the total revenue divided by the total unique visitors.

Avg $ (Average $ per Conversion) is simply the total revenue divided by the total conversions.

Conversion (Conversion Rate) is the number of conversions divided by the number of unique visitors.

Split Testing Report

The last portion of the report (shown above) are metrics that are projected based on the previous metrics.

For this website in particular, we know that it gets about 73,000 unique visitors every month, so we start with multiplying 73,000 by the conversion rate for each variable. This tell us that “assuming the website gets 73,000 visitors next month, if we switched over to this version we would generate this many conversions.”

For the next field (Revenue), we simply multiply the conversions (calculated in the previous paragraph), but the Avg $. This tell us that “assuming the website gets 73,000 visitors next month, if we switch over to this version we would generate this much revenue.”

The last field (Lift), is a simple statistic to show the difference in revenue as a percentage against the control. The math is as follows: (Revenue – Control Revenue) / Control Revenue. This tells us “if we switched to this version instead of the control, we would see an increase/decrease in revenue by X percent.

Deciding On a Winner Isn’t Easy

So let’s say you’ve run a test and generate the results shown above. How do you decide which variable wins the test? Most people would look at that data and say that since v7 showed a Lift of 23% overall it’s the best. And there is nothing wrong with that, it’s a logical conclusion based on available evidence. The Lift that I’ve shown you to calculate with this report is a variable based off all the other 8 metrics (if you reverse engineer it, it literally is derived from the original 3 metrics). So it’s a good general way to decide.

But not all tests are decided this way. Some tests get so close that measuring the Lift isn’t enough. Sometimes you could get a Lift of only 1%, which isn’t very impressive. Therefor you need to decide what priority each of the metrics plays in making your decision on the winner. So take all 9 metrics, and organize them from most important factor to least important factor.

For the company I do this for, we generally go by this decision priority list:

Lift
Revenue per Visitor
Average $/Conversion
Conversion Rate
Projected Orders/Revenue
Orders/Revenue/Visitors

However some people prefer to have a higher conversion rate rather than flat out Lift or RPV. So they might assign that as the highest priority. Always establish your decision hierarchy BEFORE you start testing, otherwise your hierarchy will be skewed because you are already seeing data and it will influence your decision process – THIS IS VERY BAD – don’t let your stats control your decision factors, they must remain independent.

Related Articles

• The Art of Split Testing – MindValley Labs
• Getting the Right Message Out – Inside Campaigner
• Split Test Case Studies – Visiolo