Rich-text Reply

Optimizely not splitting traffic evenly

jrus332 12-19-16
Accepted Solution

Optimizely not splitting traffic evenly

[ Edited ]

I've set this test to split the traffic 50/50 but it's at 70/30. Any ideas on what might cause this? The test has been running for about 4 days. Thanks! Smiley Happy

 

Screen Shot 2016-12-19 at 10.42.47 AM.png
Level 2

JasonDahlin 12-19-16
 

Re: Optimizely not splitting traffic evenly

Are you sure you *never* had this experiment running using a different percentge split? (I presume so, but, have to ask)

 

Is your QA team (or other) using a force parameter for any of their testing (especially any automation which would resutls in a significant number of users being buckected into the vartiation specifid by the force parameter)

 

Could someone have used the force parameter in creating a link to your site/page?

 

While you work to figure this out, you may want to clone the existing experiment, verfify the clone is at 50-50, then start the clone and pause the current.  That way you'll get a 50-50 split for all traffic. (This would result in users currently one of the buckets in the current experiment beign re-allocated and possibly landing in a different variation).

--Jason Dahlin
Analytics and Testing Guru Smiley Happy
Experimentation Hero
jrus332 12-20-16
 

Re: Optimizely not splitting traffic evenly

Hi Jason,

 

Thank you for the suggestion. I did as you suggested and I'm experiencing the same results. Any other ideas? 

Screen Shot 2016-12-20 at 10.58.05 AM.png
Level 2
JasonDahlin 12-20-16
 

Re: Optimizely not splitting traffic evenly

@jrus332- This is definitely odd.  I double checked our experiments and the only ones that are not 50-50 visitor split are QA versions where I explicitly set the % different so that no one would see the changes until we wanted them too.

 

Can you PM me the details (URL of the page that is potentially redirected) an dI'll see what I can tell from looking at the optimizely data object.

--Jason Dahlin
Analytics and Testing Guru Smiley Happy
Experimentation Hero
David_Orr 12-21-16
 

Re: Optimizely not splitting traffic evenly

Hi there,

 

In order for Optimizely to track a visitor, our events that contain the visitor's bucketing info(the experiment they were bucketed into) needs to go out. From what it appears, some visitors with slower connections's browsers are able to successfully send off the events prior to the redirect which is why you see 545 visitors. 

 

The reason why you are seeing this issue is because Optimizely X utilizes a browser's local storage. One limitation with local storage is that that data cannot be shared across origins. Meaning the local storage items on a HTTP page cannot be accessed on a HTTPS page. 

 

Here is a link to our KB article for more info: https://help.optimizely.com/Set_Up_Optimizely/Cookies_and_localStorage_in_the_Optimizely_snippet#Ori...

 

Variation 1 is a redirect experiment from http://example.com to http://www.exmaple.com. To resolve this, I recommend adding  .* as a regex match as an Origin in the Settings tab.  

 

Here is a link for more info on how to add this origin:

 

https://help.optimizely.com/Target_Your_Visitors/Cross-origin_targeting_in_Optimizely_X_Web#origin

 

David

Senior Technical Support Engineer
Optimizely
JasonDahlin 12-22-16
 

Re: Optimizely not splitting traffic evenly

@jrus332, I think @David_Orr is right.  At first, I thought he was wrong, but then I realized that the redirect is removing the subdomain:

Splitting Page: https://www.example.com/a/page-here

Redirect Page: https://example.com/b/page-here

 

Because the "www." is missing in your redirect, localStorage doesn't work which causes the issues David described above and I've illustrated below.

 

A- For most users, when a user is redirected, Optimizely is not recording that the user has been bucketed into that variation

B- Due to A, users who re-hit the splitting page are re-bucketed (new 50-50 split because they don't have a "decision" recorded for this user).

C- Based on timing, slower connections are able to "pass" issue A which results in them showing up in the resutls and, should they re-hit the splitting page, they remain in the experiment.

 

1- On the original page, neither the event nor the decision posts are completed before the redirect occurs (this is not surprising).  These are the two red items in the screen shot below.

console.PNG

2- Once I land on the redirected-to page, there is an "event" post (the black line in the screen shot above).

3- Now that I am on the redirected-to page, examining the localStorage, the variationMap is empty, indicating that Optimizely does not have a record of me being placed into this variation.  See screenshot below:

localStorage.PNG

4- If I clear my cookies and localStorage and re-hit the splitting page, I am now in variation 8056760179 which is detailed in the localStorage.  See screenshot below:

localStorage_control.PNG

5- After repeating the above "clear and re-enter" exercise a few times... after being redirected to the "flp" redirectory and verifying that my localStorage's variation_map is indeed empty, when I re-hit the entry page, I now remain on the spiltting page and my localStorage's variation map now contains the same information as shown in the screen shot for #4 above.

--Jason Dahlin
Analytics and Testing Guru Smiley Happy
Experimentation Hero
David_Orr 12-22-16
 

Re: Optimizely not splitting traffic evenly

Excellent breakdown @JasonDahlin!

 

A quick way to see what is happening is to open the browser's console, click here for instructions, and enter window["optimizely"].get("state").getVariationMap(); in the console. If you are redirected and the code returns an empty array or does not contain the redirect experiment ID, then it is safe to say that the localstorage info from one origin did not get passed to another. 

Senior Technical Support Engineer
Optimizely
jrus332 12-22-16
 

Re: Optimizely not splitting traffic evenly

Thank you Jason and David! I'll be implementing these changes and relaunching the test in the morning. 

Level 2