Introducing new insights to drive your decisions

by Optimizely ‎09-23-2015 September 23, 2015 - edited ‎09-24-2015 September 24, 2015

Today we're excited to announce the addition of two new metrics to your Optimizely experiment results. In the past, we've only made it possible to view unique conversions (the number of individuals who completed a goal at least once) for a given experiment goal. Now we're adding a "Totals" view to the Goal Module, which provides two new metrics you can measure as part of your experiments:

 

  1. Total Conversions: The total number of times a goal was completed, regardless of whether it was completed more than once by the same visitor.
  2. Conversions per Visitor: The average number of times a goal was completed per visitor.

 

Here's what it looks like in Optimizely:

results_totals.png

 

Now it's possible to go beyond measuring the impact of your experiment on whether users engage with a goal once, to measuring the extent to which your experiment changes the number of times your visitors engage with a goal.

 

For example, let's say you have a news website featuring a "Related Content" module below each article. You want to know whether a change to the design or placement of the module impacts the extent to which people engage with other articles. In this case, unique conversions only tell you whether your visitors clicked on a new article a single time. It doesn't tell you whether they continued to click and view even more articles, generating more impressions, page views, and ultimately higher ROI.

 

We hope these new metrics teach you more about your visitors' behavior and give you even more insights to power delightful, optimized experiences.

 

This feature is currently available on our Results Page and uses Optimizely’s industry-leading Stats Engine to calculate statistical significance with always-valid results. In the future, it will be available in our REST API.

 

We look forward to hearing what you think!

 

Thanks,

Sean

 

Does Total Counts work for my iOS and Android experiments?

Yes! It’s actually been available on iOS and Android for a couple months already.

 

Will this change how long it takes my experiments to reach statistical significance?

This will not affect the way statistical significance is calculated for unique conversions. However, you may notice that the change in conversions per visitor may reach statistical significance at a different time than the change in the unique conversion rate does.

 

Are Total Counts available in the REST API?

Not yet. We anticipate adding this feature to the REST API in the future.

 

Do Total Counts include outlier detection?

Optimizely currently does not do automatic outlier detection. If you are an Enterprise customer you can get access to raw data through our Customer Success team and remove outliers.

 

How can I access this new feature?

This feature is already enabled for all experiments that started after 5/28/15. If you've started an experiment after that date, you should see a "Totals" option in each goal module on that experiment's results page.

Comments
by cansadam
‎09-24-2015 September 24, 2015

Really excited about this new feature. You say it's currently available, but how do we access it? Thanks!

Level 1
by Optimizely
‎09-24-2015 September 24, 2015

@cansadam - Glad to hear you're excited! You should see this toggle in all goal modules for any experiment that was started after 5/28/15. If your experiment meets that criteria and you're still not seeing it, let me know and we'll look into it!

Optimizely
by cansadam
‎09-25-2015 September 25, 2015

@sean Thanks for your reply. I have experiments I started two weeks ago and I still don't see Uniques/Totals toggle pictured in your screenshot above. Is there a special view or something similar that I need to turn on? I'm happy to write a support ticket if that's the best way to look into this.

Level 1
by Optimizely
‎09-25-2015 September 25, 2015

@cansadam - Sorry about this -- there was actually an undiscovered issue preventing your account and a few others from seeing this feature. So thank you for reporting it! 

 

The issue has been resolved and you should be seeing the change now -- lmk if you're still having issues or if anything else comes up!

Optimizely
by cansadam
‎09-28-2015 September 28, 2015

Thanks, @sean, I see it now!

Level 1