Optimizely X Bug: Users being dropped from experiment variation
(I just filed this as a ticket, but would like to share since others might be getting skewed data as a result of this issue)
There is a bug in Optimizely X whereby users are being dropped from an experiment entirely once they no longer meet the Audience conditions for said experiment, causing all posterior events to not be logged and, consequently, skew the experiment results.
The user then gets bucketed into the experiment and, during the course of the experiment, gets the token:
If the user now visits the original test page (i.e homepage), he/she will immediately be removed from the test and any posterior events will no longer be logged for this user against the experiment.
This is quite a severe bug, which does not occur in Optimizely Classic, causing valuable metrics/goals to be lost and skewing the overall test results.
Attached is a screenshot showing the issue.
Please keep this in mind if you're testing anything that has a similar condition: such as starting a test for only non-paying users, then those users becoming paying members, and re-visiting the original test page, causing metrics to stop being logged.
I hope this can be resolved soon, since it's forcing us to go back to Optimizely Classic.
Hi Robert, this is not something I can control. It is a bug with Optimizely X itself, and it kicking users off experiments after being added. It only happens when the user revisits the test page after no longer meeting the Audience criteria: Optimizely X will boot the user from the experiment, causing any subsequent events not to register in Results. This is quite a serious issue, and it compromises test results.
It does not occur with Optimizely Classic and needs to be fixed at the core code level.