A/B Testing Done Right
A/B testing, a.k.a. “split testing,” is not as easy as Jackson 5’s lyric “As simple as…ABC” would have you believe. Sure, it sounds basic, but that doesn’t stop people from screwing it up and then swearing by their flawed results.
Are you testing yet? (If not, you should be.) Are you getting the most out of your tests? Read on for quick tips on A/B testing the right way.
Tip #1: Only One Variable, Please
“15 Items or Less” may work at Trader Joe’s, but it’s not good for testing. Granted, sometimes you want to study more than one variable, but unless you’re overhauling the site or rebranding, you shouldn’t do it.
The reason for my stern warning is that if you’ve tested subject line, photo, headline, and layout all at once, then how do you know which variable had the greatest impact?
This type of “too much of a good thing” experimentation also tends to lead to sweeping generalizations that all elements in the test are equally responsible for the variation’s success or failure. Obviously that’s untrue and it simply means you must test again, which begs the question: why not just do it right the first time?
Tip #2: Look at the Whole Funnel
I’m a big fan of whichtestwon.com, a cool site that offers a weekly A/B or multivariate test case. What makes the site cool is that it features a variety of tests (print, web, email) run by a wide cross-section of companies. What makes it slightly less exciting is that readers typically only see one statistic, which is good for a brief article, but bad for making business decisions.
To truly understand performance, you should be looking at a variety of numbers. Ideally your tracking codes will be monitored at every step of the process. If they’re not, well, maybe it’s time to change that.
Testing subject line and looking at open rate is a classic example of how marketers can take a myopic view. The subject line “Cheaper Than a Meal!” may get more people to open your message, but if your price point is too high, it may not generate as many orders as a version that’s upfront about your product’s cost.
Tip #3: Test Interesting Variables (and Test Them Again)
True, you can’t go wrong with a good subject line test, but don’t limit yourself to something so basic. You can also test things like photos, layout, benefit descriptions, and the size, shape and color of your call to action buttons.
Also don’t assume that just because something worked once or worked last year it will work again. Designs go in and out of fashion just like clothes, so it’s important to retest, lest you be caught wearing pastels when brights are all the rage.
Tip #4: Mix It Up With a Four-Way
Puns aside, a four-way test can help you get even more out of your testing. To conduct a four-way test, simply pick two independent variables and plan to test all of them. You can use a chart like this:
|A||Version 1A||Version 2A|
|B||Version 1B||Version 2B|
Four-ways are double the work of a basic A/B tests, and if you want accurate results, you’ll need a large audience. But if you’ve got the time and the audience, getting to test two independent variables at the same time – and getting reliable results – is pretty rad.
Tip #5: Dig Deeper for Answers
Probably the biggest mistake I’ve seen people make is to assume that the numbers tell the whole story. While data is essential, you may not have all of it. And when you dig deeper, a new story may reveal itself. Even though one email may have sold more widgets or garnered more downloads, if your tactics were shady, it may do long term damage to your brand. The lesser-performing message could also have unexpected value if it appeals to a different segment of your users.
Don’t feel boxed in by the statistics you have. Go with your gut and come up with new ways to test by segmenting your audience, soliciting feedback or simply coming up with new metrics to track.
What tips do you have when it comes to A/B testing? Share them below!
If you liked this, you might also enjoy…