Rich-text Reply
Highlighted

Understanding mixed results

mavila 4w ago
Accepted Solution

Understanding mixed results

Hello, 

 

I wasn't sure what category to put this question on. I hope I can get some help understanding some results we are getting in our experiments. 

 

Here's the typical scenario of most of our tests: 

1. Original stays the same.

2. Variation has changes in the CTA or some of the content.

3. These pages direct visitors to land in a page that is part of a e-comm funnel. 

4.  We test pageview conversion or click conversion. So if visitors land on the variation, we think the changes we made will make them convert to the next page of the funnel. or if visitors click on the button of our variation, they will convert to the next page of the funnel.

 

So for the pageview experiment, most of the times we noticed that the conversion on variation showed very low (losing variation at times) conversions but the secondary goal which is revenue, shows very high revenue in the losing variation. 

 

For the click conversion we noticed that the conversion is very high but the secondary goal which is revenue shows it very low even though the variation seems to be winning. 

 

Also I don't think it has nothing to do with one being pageview conversion or click conversion I think is just how optimizely gathers info for the revenue but I want to understand that type of result a lot more. 

 

Has anyone had this type of scenario and how do you explain this mixed result behavior?

 

Thank you in advance.

 

~ Monica
Web SEO Analyst and
A/B Testing Coordinator
Level 3

Re: Understanding mixed results

Hi Monica,

Based on the description alone, it is difficult to explain the root cause of the scenario without looking further into the actual results and experiment's implementation. I recommend you file a support ticket so technical support engineer can investigate. Instructions on how to file a support ticket can be found here - https://help.optimizely.com/Account_Settings/File_online_tickets_for_support.

In the ticket please provide the experiment id where you are seeing this behavior on the results page so we can better understand what is happening.
Optimizely
mavila 3w ago
 

Re: Understanding mixed results

Thank you. I thought maybe others had experienced the same and found a way to explain it. I will open a ticket. 

 

~ Monica
Web SEO Analyst and
A/B Testing Coordinator
Level 3