Why A/A Testing is a Waste of Time
The title may seem a bit controversial, but this is a fairly common question I get from large (and small) companies – “Should I run A/A tests to check whether my experiment is working?”
I’ve been doing split and multivariate tests since 2004, and have watched nearly every one like a hawk. I I’ve personally made every test cockup in the book and wasted many days on fruitless effort, in order to become better and to continue to hopefully improve my ability to run valid tests. What does my experience tell me here?
There are better ways to use your precious testing time
Firstly I don’t want to come across as saying running an A/A test is wrong – just that my experience tells me that there are better ways to use your time when testing. The volume of tests you start is important but even more so, is how many you *finish* every month and how many from those that you *learn* something useful from. Running A/A tests can eat into ‘real’ testing time.
And the trick of a large scale optimisation programme, is to reduce the resource cost to opportunity ratio. To ensure velocity of testing throughput …read more