Rich-text Reply
Highlighted

A/A Testing

Jonny 3w ago

A/A Testing

Hello,

 

I recently completed an A/A test using a control and an identical experiment. The purpose of which was to test the accuracy of the experiments and gauge whether future tests would be reliable. The test included 22,000 page views, and despite being identical, the experiment generated a conversion rate that was 11.3% lower than the control. Is this a common issue?

Level 1

Re: A/A Testing

Were the results significant?

--Jason Dahlin
Analytics and Testing Guru Smiley Happy
Experimentation Hero
Shlomo a week ago
 

Re: A/A Testing

I wouldnt consider it an issue unless the test results were significant.
What was your baseline conversion rate?
How many visitors to each variation?
Did you check GA segmentation to ensure the traffic quality was split evenly? ( Assuming you run on multiple channels)
Level 2