Christian Little

Five days without posting, yeah I suck. But in those five days I’ve been to a wedding where my wife was the maid of honor, and the MV testing demonstration I promised earlier took a major turn and crashed horribly, which I’m going to discuss today so you can avoid this problem.

I posted some information last week about doing a transparent multi-variant experiment using Google’s Website Optimizer.

After over a week of problem solving and trying several solutions, we called it quits.

What Happened?

Google’s tool is very handy if the visitor never leaves your website during the conversion process. However, it sucks donkey balls if you try to run the MV test across multiple domains during the conversion process. The website we tested the platform on has a front end website that discusses the products in question, but when the customer wants to make a purchase they are taken to a secure transactional server where the actual purchase happens.

Don’t ask me for the reasons why we do it this way, as I don’t know. The company has been doing it long before I came along, and it works for them.

So we setup the test and had nothing but technical failure after failure. For those of you that want to do any form of testing across multiple domains, forget using Google. Here’s some of the problems you will encounter:

• The multi-domain code that Google gives you for this situation is a hack, it’s not actually completed code. They basically give you a few different pages of coding showing you how to modify their core single-domain testing code.
• In addition to hacking up the basic javascript, you also have to add code to the page where people land on the second domain. This step was actually straight forward.
• On the page that sends people from your first domain to your second domain, you have to add special “onclick” commands to the links that send people to the second domain.

So we spent a day figuring out the new code and implementing it on our test site, and it worked so we pushed the code to our live server and the test began (woohoo!).

It Didn’t Update for 5+ Hours

That was our first problem. We pushed all this code onto a site and had to sit on our hands and wait. We couldn’t even tell if the data was actually being received by Google, nevermind if it was tracking everything properly.

So we waited, and discovered a few minor JS bugs that we had to correct and restart the test, which required more waiting. This happened twice in a row, and I don’t blame Google for that, we just missed a few things.

It Broke the Cart…

This is where I blame Google. If you follow their instructions for multi-domain testing, it requires you to add some onclick code to all links to the second domain.

We missed this step initially, so we had to stop the test and put this new code on our test server. It worked, great! We moved it to the live server…

And it worked…for 14 hours…

After 14 hours of running just fine from what we could tell (again, we had to wait for the stats to update, which take 6-10 hours). So everybody went home for the day thinking we’d check the stats in the morning and get an early impression of how the test was going.

WRONG.

We login to the reports and it shows no conversions. We figured it was just another piece of code we missed, so we started looking over the setup instructions again.

Then we came to an awful truth – the onclick code that was require for the links sending people to the checkout server failed. Actually it wasn’t a failure, a better word to use would be kick in the groin. When the onclick code failed, the links stopped working…which meant that our customers couldn’t checkout.

We lost about $11,000 in revenue for the 7ish hours that the site stopped allowing checkouts.

I still don’t have an explanation as to why it failed. It was working just fine, and then for some mysterious reason it just stopped working and refused to let anybody else through the site.

As you can imagine, that didn’t go over very well with the powers that be, so we stopped the experiment, ripped out the code, and started chanting very nasty things about Google for the rest of the day.

So Now What?

Fear not! I’m still going to post a complete test. However after several days of research, we are going with a more reputable testing platform. The new test won’t be ready for a few days though, as the new solution is a bit more involved when first setting up.

I’ll be posting the details on the new test in a few days.

Lesson from this Experiment

I still believe that Google’s Website Optimizer is a great tool, unless your visitors have to leave your site during the conversion process. If you send you visitors to a payment processing site or anything like that (I don’t think you could even use Website Optimizer if you use Paypal…that would be a nightmare).

Previous Posts for This Experiment

• Initial Announcement of the Test
• Launch Day – Showing the Details

Related Blogs
None. Normally I would post links to related posts to other blogs, but I couldn’t find any related to this problem. That means either we totally messed up the coding, or nobody has tried using Website Optimizer across multiple domains and posted their experiences. Either way, I’m too pissed off at Google to give a damn.