- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Why I have trust issues...
Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Running an experiemnt on our site now that is a 50/50 split between original and a variation. However, the variation is empty; no changes were made.
Interesting stuff :-)

Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
- Tom Fuertes | CTO @ CROmetrics / LinkedIn
"Most Impactful Use of Personalization" and "Experience of the Year" Optie award winner.
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report

It looks like your total visitors in the first screenshot is 3,685 while total visitors in the second screenshot is 3,639, so I think that the timeframe of the data between these two is slightly off. Maybe it'd be more accurate if you check again once the duplicate has gone on for a full 24 hours, so you can compare midnight to midnight for both.
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Yesterday's results...
264 orders to 255 orders: 10 order difference.
Side note: There were 240 orders recorded in Google Analytics
7,505 users to 7,442 users: 63 user difference
+7.99% improvement to -4.02% improvement: 12.01% difference
I am not saying these numbers should be exactly the same but it certainly means to me that Optimizely results should be taken with a grain of salt and should be a part of a larger conversation for sure.

Re: Why I have trust issues...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report
Small numbers suck, especially for low-power experiments.
It doesn't look like either of these have reached "significance" yet. At best, anything that has not achieved significance should be treated as "directionally positive/negative". If you let them run longer, the epxectation is that these A/A tests would eventually settle on "no directionality".
Analytics and Testing Guru



