I was interested to hear that the Huffington Post – the technorati top rated US Blog is using real-time testing of popularity to work out which headlines to use some of their more important pages. The website randomly shows one of two headlines. The version with the most clicks becomes the final version. For a website with this much traffic they find that five minutes is enough time to find a significant difference. But the point is that if they do enough experiments Huffington post will become the world experts in what makes an effective headline!
Companies such as Amazon, Google and Intuit have used such A/B real time testing to user behaviour on versions of their websites for some time. They have found that small differences can lead to big changes in the way people behave.
Such testing led Google to refine exactly how much white space to put around its logo and to change from putting a pale blue background behind its ads to putting a pale yellow background (actually this fits in with reading research – pale yellow provides the best contrast for people to read). They also found that increasing the number of searches listed to 30 resulted in longer search times which meant people didn’t hang around waiting for the results. Amazon found that making a coupon code to prominent on a website lead to much lower uptake rates. People were were thinking “better go and find a coupon” instead of actually proceeding with a sale. Ron Kohavi of Microsoft Research has used multiple level test designs and studied interaction effects to good effect.
Jakob Nielsen , Usability Guru points out that whilst more qualitative usability tests are better for finding the really big problems with your website – A/B testing does have its place.
- It is great for measuring actual behaviour in context (a point I made about the value of social media feedback in general yesterday).
- It can measure very small differences (because you can test on a large number of users again as I mentioned yesterday).
- It can resolve trade-offs between conflicting guidelines
- It is very cheap!
Jakob argues that this doesn’t necessarily lead to understanding of why? However I would suggest that if you do lots of studies then over time any analysis leads to deeper understanding.
I guess what interests me about this is that it really is very hard to guess what a good online design or campaign is going to be. In some ways this is why social media is a bit scary. Time and time again doing user testing I would be surprised by the results I got from the data. Our intuition is poor.
Why could such quantitative testing be important for people thinking about social media? Well we can quickly test our hunches by setting up a hypothesis and collecting data. By doing this we can learn about what really works for your particular context. I have read more and more posts entreating marketers to be more experimental. If you were to experiment in a measurable way then you can start to build up your understanding of what works too!
SociaLNK is keen to support you as you start to experiment with social media. Contact us.
