Rich-text Reply
Highlighted

Understanding mixed results

mavila 03-25-19
Accepted Solution

Understanding mixed results

Hello, 

 

I wasn't sure what category to put this question on. I hope I can get some help understanding some results we are getting in our experiments. 

 

Here's the typical scenario of most of our tests: 

1. Original stays the same.

2. Variation has changes in the CTA or some of the content.

3. These pages direct visitors to land in a page that is part of a e-comm funnel. 

4.  We test pageview conversion or click conversion. So if visitors land on the variation, we think the changes we made will make them convert to the next page of the funnel. or if visitors click on the button of our variation, they will convert to the next page of the funnel.

 

So for the pageview experiment, most of the times we noticed that the conversion on variation showed very low (losing variation at times) conversions but the secondary goal which is revenue, shows very high revenue in the losing variation. 

 

For the click conversion we noticed that the conversion is very high but the secondary goal which is revenue shows it very low even though the variation seems to be winning. 

 

Also I don't think it has nothing to do with one being pageview conversion or click conversion I think is just how optimizely gathers info for the revenue but I want to understand that type of result a lot more. 

 

Has anyone had this type of scenario and how do you explain this mixed result behavior?

 

Thank you in advance.

 

~ Monica
Web SEO Analyst and
A/B Testing Coordinator
Level 3

Leandra_Kim 03-26-19
 

Re: Understanding mixed results

Hi Monica,

Based on the description alone, it is difficult to explain the root cause of the scenario without looking further into the actual results and experiment's implementation. I recommend you file a support ticket so technical support engineer can investigate. Instructions on how to file a support ticket can be found here - https://help.optimizely.com/Account_Settings/File_online_tickets_for_support.

In the ticket please provide the experiment id where you are seeing this behavior on the results page so we can better understand what is happening.
Optimizely
mavila 04-02-19
 

Re: Understanding mixed results

Thank you. I thought maybe others had experienced the same and found a way to explain it. I will open a ticket. 

 

~ Monica
Web SEO Analyst and
A/B Testing Coordinator
Level 3

Re: Understanding mixed results

Hey there, 

Just wanted to reach out to you and let you know that my team has also encountered odd results in nearly exactly the scenario you described. 

What we did to get a better idea at what was going on with conversion was added up all visitors to buyflow in A and B of test and measured buys against that number of visitors instead of the visitors to the primary page with the cta on it. 

here is a calculator to enter your info. https://www.surveymonkey.com/mp/ab-testing-significance-calculator/

The tricky thing is, we dont know if the change in copy effected them to be more willing to buy in the end, the tests say yes. But its not a clean clear answer, Its actually a relief to see that other people are getting these results too.  Best of luck to you