Rich-text Reply

Optimizely X Bug: Users being dropped from experiment variation

ralphandrew 10-25-16

Optimizely X Bug: Users being dropped from experiment variation

(I just filed this as a ticket, but would like to share since others might be getting skewed data as a result of this issue)

 

There is a bug in Optimizely X whereby users are being dropped from an experiment entirely once they no longer meet the Audience conditions for said experiment, causing all posterior events to not be logged and, consequently, skew the experiment results.

 

As an example, say you have an experiment that activates on the homepage, with an Audience that specifies a Custom Javascript condition to include users that do not have a specific localStorage token:

 

!localStorage.getItem("access_code")

The user then gets bucketed into the experiment and, during the course of the experiment, gets the token:

localStorage.setItem("access_code", "true")

If the user now visits the original test page (i.e homepage), he/she will immediately be removed from the test and any posterior events will no longer be logged for this user against the experiment.

 

This is quite a severe bug, which does not occur in Optimizely Classic, causing valuable metrics/goals to be lost and skewing the overall test results.

 

Attached is a screenshot showing the issue.

 

Please keep this in mind if you're testing anything that has a similar condition: such as starting a test for only non-paying users, then those users becoming paying members, and re-visiting the original test page, causing metrics to stop being logged.

 

I hope this can be resolved soon, since it's forcing us to go back to Optimizely Classic.

Screen Shot 2016-10-24 at 5.04.26 PM.png
robertchan 10-25-16
 

Re: Optimizely X Bug: Users being dropped from experiment variation

Couldn't you drop a cookie on visitors that would then keep their history persistent with whatever variants you're running?
Robert Chan

Experimentation Hero
ralphandrew 10-25-16
 

Re: Optimizely X Bug: Users being dropped from experiment variation

[ Edited ]

Hi Robert, this is not something I can control. It is a bug with Optimizely X itself, and it kicking users off experiments after being added. It only happens when the user revisits the test page after no longer meeting the Audience criteria: Optimizely X will boot the user from the experiment, causing any subsequent events not to register in Results. This is quite a serious issue, and it compromises test results.

 

It does not occur with Optimizely Classic and needs to be fixed at the core code level.

robertchan 10-26-16
 

Re: Optimizely X Bug: Users being dropped from experiment variation

Okay, I would file a ticket with them and I'm sure they'll address it asap.
Robert Chan

Experimentation Hero