Rich-text Reply

Strange experiment results. Any advice?

romanp 12-26-15

Strange experiment results. Any advice?

I'm running two experiments:

- The first one check hyposesis that hiding CTA improves convesion on the first step of sign-up funnel

- The second is checking conversion on second step of the sign-up funnel

 

1) In the first experiment I added variance which hides one of four CTA buttons.

I have two goals in experiment 

- Goal 1 - Clicking on one of four CTA buttons

- Goal 2 - Clicking on one of three CTA buttons (the same buttons exclusing the one I hide in experiment).

 

I expect the Goal 1 will always have higher conversion rate than Goal 2 but in one variance conversion rate of Goal 2 is higher.

What could be the reason for this?

 

2) In second experiment I try to measure conversion on the next step of sign-up funnel.

I have just one variance and the goal is button click on sign-up page. To exclude all runnig campaigns redirecting  to sign-up page I defined experiment URL targeting as

www.mysite.com/signup?src=homepage_cta1

www.mysite.com/signup?src=homepage_cta2

www.mysite.com/signup?src=homepage_cta3

www.mysite.com/signup?src=homepage_cta4

 

which are links called by CTA1..CTA4 on home page.

 

Let's assume that there are no other pages/campains using URLs above.

I'm courious why I get more visitors in this experiment than conversions in previous experiment.

 

What could explain the discrepancy? 

 

Thanks!

 

Level 2

David_Orr 12-28-15
 

Re: Strange experiment results. Any advice?

Hi Romanp,

 

Thank you for your post.

 

Experiment 1:

 

Do you believe that the buttons are incorrectly being tracked? May I ask if you were able to verify whether the correct goals were being fired? Here is an article that will walk you through on how to troubleshoot goals:

 

[Checking for whether goals are firing correctly](https://help.optimizely.com/hc/en-us/articles/200063579-Troubleshooting-Goals-don-t-track-correctly)

 

Experiment 2:

 

 

Please correct me if my understanding is incorrect. 

 

* Step 1 : homepage.html (Experiment 1 runs on this page)

* Step 2: signup.html?src=homepage_1 (experiment 2 runs on this page)

 

What we are experiencing is that Optimizely is receiving more visitors on step 2 (Experiment 2) than step 1 (experiment 1).  

 

Here are the settings I would check first if I were to look at the experiments and site:

 

* Make sure both experiments started at the same time, or within minutes depending on the site. 

* Make sure both experiments are new and results were not reset. This will cause previously bucketed visitors to show up in the results. 

* Traffic allocation settings.

 

If you are still running into issues, we can troubleshoot this via email. Let me know if you would like to take that route and I'll send you an email to the address in your profile.

 

David

 

Senior Technical Support Engineer
Optimizely
romanp 12-29-15
 

Re: Strange experiment results. Any advice?

Hi David,

Thanks for answers.

1) I don't know if it possible to track buttons incorrectly when I just select them when defining goals. I will check troubleshooting article.

2) Experiments were not newly created. I just count the difference in new fired events. I will create a new experiments.

Thanks

Level 2
JDahlinANF 12-29-15
 

Re: Strange experiment results. Any advice?

Question 1
"In the first experiment I added variance which hides one of four CTA buttons.

I have two goals in experiment 

- Goal 1 - Clicking on one of four CTA buttons

- Goal 2 - Clicking on one of three CTA buttons (the same buttons exclusing the one I hide in experiment).

 

I expect the Goal 1 will always have higher conversion rate than Goal 2 but in one variance conversion rate of Goal 2 is higher.

What could be the reason for this?"

 

Easy answer - given fewer options, users are finding it easier to make a choice.

 

Question 2

"Let's assume that there are no other pages/campains using URLs above.

I'm courious why I get more visitors in this experiment than conversions in previous experiment."

 

This ought not happen.  I presume you've double-checked the URL targeting?

I've seen similar happen when a user clicks a button that submits a form.  The click event only captures clicking the button-click and not the actual form-submit (like a user pressing [Enter]).  So if the form submit causes the URL param to be added and there is some way to submit the form without clicking the submit button, what you see would be the result.