5 Uncomfortable A/B Testing Questions
Testing is supposed to be straightforward and extremely transparent. It should be so easy to see the ROI – especially when compared to opaque stuff like SEO. But is it really so transparent as we’d like to think?
There are some uncomfortable questions about transparency, but we can answer them. In this post I’ll address the top 5 “uncomfortable questions” along with comments from fellow optimizers.
1. Is the lift still there?
You ran a test, and had a solid winner – the sample size was adequate, test ran for 2 business cycles, confidence level was above 95% – and the lift was like 10%. You’re confident in the result – and implement the winner on your site.
Now 3, 6 or 12 months go by. Is the lift still there? How would you know?
Let’s say your boss or client asks that question. What would you say?
You can’t rely on web analytics data – and just compare time periods – as everything changes. Your traffic sources, external environment and so on – it wouldn’t be apples to apples comparison, too many uncontrolled variables.
Yes, re-testing it seems to be the easy way out, but in reality it might not be so easy. Testing something …read more