Christian Little

Some of you will recall my dissatisfaction with AB testing with Google last month, which was due to a large number of technical problems we ran into when setting up their code. Suffice to say, it was a horrible failure and we crashed a site for 14 hours when we put their code in, as you can imagine we were not impressed and threw their platform out the window.

If you want the full story, you can read the drama here:

1. Original announcement about demonstrating an AB test
2. AB test launched with Google Website Optimizer
3. Where it all fell apart

Since then, I went looking for some of the other options for testing platforms and spent some time with each of the major players:

• Offermatica – this is the testing platform owned by Omniture, who owns Site Catalyst. That would have been ideal since we use Site Catalyst and the accounts are interchangeable. The biggest downside was the price, we could feed a third world country for what they wanted to setup testing on one website.
• Site Spect – this is a very powerful platform that has some very cool feature. They gave me a demo and I was blown away and left speechless. It’s amazing what they’ve built, but there were some problems that prevented us from moving forward – cost being one (it wasn’t as pricey as Offermatica though, and technical issues (they wanted us to change the DNS on our domains to point to their servers which we are not prepared to do).
• Vertster – this platform didn’t offer as many features as the other two, but it showed promise and offered the type of reporting we were looking for. The price was fair, and they didn’t require any major technical shifts on our end to get setup.

Trial Period

One of the other major boons that we liked about Vertster was they offer potential clients a 2 week trial to use their software (none of their competitors offer this). We figured with an offer like that we couldn’t lose, so we signed up on a trial account and setup an AB Split Test on one of our websites.

Setting up the test in their platform took about 2 minutes, you just tell it the date and URLs to send the traffic to. After that, it gives you several pieces of Javascript (the rotation code which actually sends your traffic to the tests, the landing page code, and the conversion code). All in all, it took about 15 minutes for us to insert the JS code on our site and verify that it worked.

This is one of the reports they provide you with (yes this was our test data). It shows you the important information for each variable, which we called Control and Homepage v2 (we were testing the impact of a different homepage design on our website). As you can see, the Conversion Rate (Sales CR) and Average Order Value (AOV) are the most important elements in this report, as the revenue and orders will never match up since we only sent 15% of our traffic to the test site.

Technical things to be Aware Of

In my adventures with Google Website Optimizer, one of the problems we had was that our storefronts and our transactions are handled by 2 different servers (it’s actually 6 servers, but we’ll just pretend it’s 2). When a customer first comes to our site, they see a nice looking website that shows the product and gives all the information. When they decide to purchase, they get sent to a different website to process the transaction (same idea as sending somebody to Paypal).

With Google’s tool, this failed horribly because they need you to add a bunch of javascript code when the user travels from one domain to another, which just didn’t work. I’m not that surprised, the code they sent us was a hack of their code and it says very clearly on the site that it wasn’t built for this.

With Vertster, they use 1st party cookies which requires your visitor start and end on the same domain. This was a problem for us, as our visitors never return to the original site (just like Paypal rarely returns customers to your site after they purchase). So we setup an iframe that loads a page from our information site which has the conversion code. It’s the same idea that most affiliate networks use, except instead of an image we used an iframe (I suspect we could have gone either way on that).

Other things to Know About Vertster

When I first talked to Vertster, I was directed to a man named Scott Miller, who has been our sole contact for the entire process. He has been absolutely amazing and answered every stupid little question I would email him within a few hours. What I didn’t know at the time was that he was the CEO of Vertster, which I discovered by accident when I watched some presentations that he gave at SES about their platform. He never once flogged his title in front of me (I’m just a marketing peon afterall), and handled us better than I’ve seen by most companies that manage clients. So kudos to him!

The other thing to be aware of is that Vertster is NOT a free platform. They’re also not the most expensive platform. I don’t think I’m allowed to release their pricing information here, but I can put it in a scale like this:

Google Website Optimizer – about the equivilant of McDonalds food.
Vertster.com – about the equivilant of eating at a nice family restaurant.
Site Spect / Offermatica – about the equivilant of eating at a 5 star restaurant where they kick you out for not wearing something fancy.

To put it in other terms, Vertster costs about 60% less than Offermatica and Site Spect charge. However there is the drawback that those two services offer a huge amount of more features.

The other kickass factor about their platform is that it is independent of domains, meaning you can run their software on any website without extra charge. You pay based on the volume of traffic, not a per-domain basis that Site Spect and Offermatica use.

I’m happy to say that because of all this, we signed a 12 month contract with Vertster and look forward to many more tests in the future.

Getting Started

If you want to try out Vertster, they are still offering the 14 day trial on their website.