(I just filed this as a ticket, but would like to share since others might be getting skewed data as a result of this issue)
There is a bug in Optimizely X whereby users are being dropped from an experiment entirely once they no longer meet the Audience conditions for said experiment, causing all posterior events to not be logged and, consequently, skew the experiment results.
As an example, say you have an experiment that activates on the homepage, with an Audience that specifies a Custom Javascript condition to include users that do not have a specific localStorage token:
!localStorage.getItem("access_code")The user then gets bucketed into the experiment and, during the course of the experiment, gets the token:
localStorage.setItem("access_code", "true")If the user now visits the original test page (i.e homepage), he/she will immediately be removed from the test and any posterior events will no longer be logged for this user against the experiment.
This is quite a severe bug, which does not occur in Optimizely Classic, causing valuable metrics/goals to be lost and skewing the overall test results.
Attached is a screenshot showing the issue.
Please keep this in mind if you're testing anything that has a similar condition: such as starting a test for only non-paying users, then those users becoming paying members, and re-visiting the original test page, causing metrics to stop being logged.
I hope this can be resolved soon, since it's forcing us to go back to Optimizely Classic.
