Rich-text Reply

How to analyze performance of a whole project with a bundle of pages?

tracy 06-09-14
Accepted Solution

How to analyze performance of a whole project with a bundle of pages?

I have 30 plus pages in a project but with a same goal to test, and now I'd like to give one report to show the results of these pages in the project as a whole. How can I integrate all the results through Optimizely?

If there's an access to show the project performance, that will give a great help to me!!

 

 

Tracy 

Level 2

adzeds 06-09-14
 

Re: How to analyze performance of a whole project with a bundle of pages?

Could you not use Google Analytics 'Custom Variables' to monitor performance across your website against your goals.

 

This should be pretty simple if you use the Optimizely Google Analytics integration. Simply open the report in Google Analytics and filter it by your chosen goals to see how your test variations are performing against those goals.

 

David Shaw
Level 11
Gary 06-09-14
 

Re: How to analyze performance of a whole project with a bundle of pages?

Hi Tracy,

This is not currently possible within Optimizely to compare a goal's performance across multiple experiments as you describe. The only way to do this, would be to open the results page for each experiment, which sounds incredibly tedious, and not scalable for 30 experiments.

One reason for this is each experiment influences the goal differently, depending on what you changed; due to this, comparing the data would not be a direct comparison (think apples to oranges rather than apples to apples).

What goal are you trying to analyze here and what are you hoping to understand by comparing the performance across all of the experiments?
- Gary
Optimizely - Turning the World's data into action
Optimizely
tracy 06-10-14
 

Re: How to analyze performance of a whole project with a bundle of pages?

Hi Gary,

The experiments are executed to compare performance between two kinds of landing pages, one is product listing page and the other one is homepage. I set up a group of product listing pages and the variations are all the same, the homepage. That's why I hope to analyze them as a whole.

Do you think their goals are different?

Google Analytics cannot help because I cannot separate special visits to homepage...
Level 2
ryanlillis 06-10-14
 

Re: How to analyze performance of a whole project with a bundle of pages?

[ Edited ]

Hi @tracy ,

 

Based on my understanding of your request, I have a few suggestions depending on what your ultimate goal is.

 

  1. Monitoring Campaign: If your goal is to see the cumulative effect over time of testing as a whole (in other words, the effect of both the variations you’re running and your originals) on that final goal, you can run a monitoring campaign that simply tracks that goal and see how it changes. A monitoring campaign is an experiment that isn’t actually testing anything but is being used to measure a goal. It’s a way to use your testing platform as a way to capture basic analytics. 

    All you need to do is create a new experiment, set up the goal you want to track and start the experiment.

  2. Offline Calculations: If your goal is to see the cumulative effect of all the best performing variations in the experiments you already ran, you can use an offline statistical significance calculator. Using this, you would add all the visitors who got into all your original versions and enter that number in a “Control Group/Visitors” cell. Do the same for all your variations. Then do the same for all your originals’ conversions, and all your alternates conversions. 

    Doing this will give you a theoretical look at the cumulative impact of all your variations. I say “theoretical” because as separate experiments that may have run on different dates, and, with users who saw the original in one experiment and may have seen a variation in another, we cannot know the true cumulative effect of all the best performing variations. Nonetheless it gives you a reasonable starting place to understand the impact.

  3. New Multi-page Experiment: Your last option would be to combine the pages you had successful versions on into a multipage experiment (it would be quite large however if you want to do it for all of them). This would essentially be a way to retest all the things you tried separately and use one timeframe, and ensure that any individual user only saw either the “full original” experience across all your pages, or the full “variation” experience across all your pages.

    This option is is essentially the same as the offline calculator option but by running an actual test, you would remove the “theoretical” aspect I previously described.
Optimizely
tracy 06-11-14
 

Re: How to analyze performance of a whole project with a bundle of pages?

Thanks a million for your solutions!
I like the 3rd solution, but have to balance the earns and pays between the 2 and 3.
BTW, could you tell me how to create a multi-page experiment?
Level 2
ryanlillis 06-11-14
 

Re: How to analyze performance of a whole project with a bundle of pages?

Hi @tracy ,

 

Setting up a multi-page test is fairly straightforward and we have an Optimizely Academy activity that details exactly how to do that. Once you run one multi-page tests you'll probably think of many use cases where it may be helpful.

 

Please note: Multi-page tests can only be done with either a Gold or Platinum account. 

Optimizely