Rich-text Reply

How to track whether a goal success is because of the experiment-variation?

andreasbalzer 04-25-16

How to track whether a goal success is because of the experiment-variation?

Hi,

lets say we host a large store to sell pencils. Our store has landing pages, pages showing categories and pages showing the actual products. Lets say we add a web banner to one of our landing pages that shows a blue pencil. How can we identify whether a user is more likely to buy pencils because they clicked on the banner on the landing page? How can we identify whether a user is more likely to buy blue pencils because they clicked on the banner showing blue pencils on the landing page?

 

Simply having a goal on the buy button wouldn't work, because that is sort of independent of the banner, right?

My current approach is to use dimensions to track banner clicks (yes/no) so it is possible to segment buy button clicks based on the dimension. But that is getting messy pretty quickly.

 

It would be great to see inbuild support for this, like to make goals dependant from each other.

 

Thanks,

Andreas

Andreas
Lyndsey 04-26-16
 

Re: How to track whether a goal success is because of the experiment-variation?

Hi Andreas,

You can use a couple techniques to set up a test that will give you the transparent results you are looking for. Creating a test using Conditional Activation, activating the experiment on the banner click, would help you to scope your experiment to only include visitors who have interacted with the page that way. You'll then be able to determine any lift in purchases for that specific group. Here's more information on Activation Mode: https://help.optimizely.com/Build_Experiments/Activation_Mode:_Activating_an_experiment_dynamically_...

You may also want to build a funnel experiment, with one variation that has the banner and one that doesn't. You can then interpret any differences between the two groups and how they interact with the page based on your goals: https://help.optimizely.com/Build_Experiments/Multi-page_funnel_experiments

Here's another great reosurce on interpreting results, which might also be helpful: https://help.optimizely.com/Analyze_Results/Notes_from_the_field%3A_Interpreting_patterns_in_your_re...

Hope that helps!



Thanks,
Lyndsey
Technical Support Engineer
Optimizely
Optimizely

Re: How to track whether a goal success is because of the experiment-variation?

[ Edited ]

Hi Lyndsey,

thanks for your reply. Unfortunately I think I don't understand it.

"Creating a test using Conditional Activation, activating the experiment on the banner click, would help you to scope your experiment to only include visitors who have interacted with the page that way."

So I would have one experiment A with my banner and then another experiment B on the purchase page that gets conditionally activated when someone clicked the banner in experiment A. As far as I know experiment B would then only provide me a number of purchases but lack a comparison to the baseline (improvement), correct? So I would run another experiment C that activates for all people who did not click that banner in A. Could I then share goals of B and C with A, so the reporting of A tracks all goals / variations? I only know I can use multiple experiments to influence dimensions but I haven't tried with goals yet.

 

"You may also want to build a funnel experiment, with one variation that has the banner and one that doesn't." That would only tell me people who may have seen (not guaranteed) the banner behaved differently. It wouldn't provide data as to whether the interaction with the banner caused the difference or whether an external variable (like a newspaper article leading people directly to the purchase page) influenced people.

 

A generic example by Optimizely on a purchase flow like this would be great.

 

Thanks,

Andreas

 

 

Andreas
Lyndsey 04-26-16
 

Re: How to track whether a goal success is because of the experiment-variation?

Hi Andreas,

You can create new content and then use the clicking / interaction with that as the basis for conditional activation, so you wouldn't need to split that into 2 separate tests. You can use this test to measure against your normal purchase flow with no banner - ie, did the purchases lift for the specific group? You can also create a variation for a non-conditionally activated experiment that has the banner, while the original doesn't. You might then also drop a cookie in the variation code once visitors click on the banner, and segment based on that cookie. You'd then be able to determine not only purchases between banner vs. no banner, but also those who actually clicked the banner and their purchase level - I'd encourage you to read some of the Knowledge Base articles on setting up experiments to determine which method will suit your needs the best.

As for funnel testing examples, here are a few:
https://blog.optimizely.com/2014/01/17/split-tests-for-the-checkout-funnel-urban-outfitters/
https://help.optimizely.com/Ideate_and_Hypothesize/Testing_ideas_for_E-commerce_and_retail_sites

And more resources are available by searching within the Optiverse.

Lastly, unfortunately you cannot share results across experiments.
Thanks,
Lyndsey
Technical Support Engineer
Optimizely
Optimizely