A/B testing is nothing more than performing a test of two elements and measuring the response. It’s called A/B testing because you are likely testing two elements against each other, such as element “A” and element “B” to see which element increases the liklihood of conversion. These elements could be a “50% off” header versus a “Buy 1 get 1 free” header. When performing an A/B test, it’s critical to hold all other elements on the page constant, to enable accurate measurement of the results.
Multivariate Testing, on the other hand, is useful for testing multiple elements on a webpage, for example, to determine the most successful combinations. When setting up and managing a multivariate test, it’s possible to test 3, 4, 5 or even more elements on a page at an assigned or random interval. The benefit of multivariate testing is that it’s possible to determine optimimal combinations, that might take hundreds of A/B tests to arrive at the same result. The negative side is that you need hundreds, if not thousands, of conversions to get an accurate picture of the algorithmic combination that has led to the best result.
In the last week, a friend of mine has been digging into multivariate versus a/b testing. He’s using Google Website Optimizer to setup multivariate testing for his e-commerce website. I like the Google system, since it’s free and provides the basics needed for good A/B or Multivariate testing. SiteSpect and Omniture have good technology as well, but they can’t match Google’s price of Free. Anyway, while he was setting it up I was giving him some advice on different elements to test. On the same day, I happened to notice an A/B test on another website (probably just because it was on my mind). PRweb, which is an online distribution source that we often use for Optimized Press Releases seems to be doing some interesting A/B testing on their registration process. Below are their two versions, let me know what you think:
PRweb-testA
PRweb-testb