Why I have trust issues...

[#1]

Running an experiemnt on our site now that is a 50/50 split between original and a variation. However, the variation is empty; no changes were made. 

Interesting stuff :-)

[#2]

That's why you have to look at a test from all segments and run the revenue through the quartile export (might be classic only)!

[#3]

We always have 2-way, 3-way, and 4-way empty tests going on to measure random noise. When we get significant results from a test, we try to look at the same timeframe as the test duration in our random noise detector tests. It helps us see additional context, especially because our site experiences a lot of seasonality! Not an exact science, but setting up ongoing empty tests can be helpful the next time your results page looks like this.

[#4]

That is a great idea. I never thought about keeping these things running all the time. Totally doing that.

[#5]

I actually duplicated the empty test so now I have two running at the same time and there are even differences between the two for the same time period. Definitely going to use this data when making decisions.

[#6]

 

[#7]

Hm, I'd guess that the discrepancy in the total orders across the two variations comes from different timing when the data was updated. I can see that you set both to 8:35 AM, but I've always been skeptical of the accuracy of that timestamp

It looks like your total visitors in the first screenshot is 3,685 while total visitors in the second screenshot is 3,639, so I think that the timeframe of the data between these two is slightly off. Maybe it'd be more accurate if you check again once the duplicate has gone on for a full 24 hours, so you can compare midnight to midnight for both.

[#8]

I agree. I'm going to let it ride and then check again tomorrow. I'll try to post the final results...

[#9]

Yesterday's results...

264 orders to 255 orders: 10 order difference.
Side note: There were 240 orders recorded in Google Analytics

7,505 users to 7,442 users: 63 user difference

+7.99% improvement to -4.02% improvement: 12.01% difference

I am not saying these numbers should be exactly the same but it certainly means to me that Optimizely results should be taken with a grain of salt and should be a part of a larger conversation for sure.

[#10]

Small numbers suck, especially for low-power experiments.

It doesn't look like either of these have reached "significance" yet.  At best, anything that has not achieved significance should be treated as "directionally positive/negative".  If you let them run longer, the epxectation is that these A/A tests would eventually settle on "no directionality".