Known Issue: Partial visitor event logging outage
On Monday, we identified a experiment issue that prevented subset of visitor events from being logged properly in Optimizely between 12:17 PM to 1:40 PM (Pacific time). As a result, customers will see lower-than-actual counts of unique visitors and revenue during this time. We know that our customers trust us with their data, and we take that trust very seriously. We’re sorry.
We want you to be fully aware of the issue and let you know that it will take you up to two additional hours for any experiment to reach the same level of significance. All experiments and variations were impacted equally, so the conclusions drawn by your experiments are still valid.
During the incident, we Optimizely had reduced ability to log visitor and revenue data. We’ve determined that up to 54 percent of the sent data that would have been logged during that time period will not be recoverable. A set of events were recovered and were logged in between 5:00PM PST and 1:23 AM PST. As a part of that process, a very small amount, 0.2%, of revenue events from a later period in the same day, Monday, 5:00PM PST to Tuesday 1:23 AM PST, were also removed as a precautionary measure to ensure the fidelity of customers’ data.
During a routine investigation to troubleshoot a not-customer-facing issue, we put an unintended strain on our backend systems, leading to this temporary failure.
Recovery and remediation:
Upon discovery of the issue, the problem was fixed within an hour and we were able to prevent any additional loss of event data. In order to prevent recurrence of this issue, we will be changing the way we do internal troubleshooting and tracking more closely the performance impact of internal maintenance and debugging tasks.
The quality and accuracy of your experiments is our top priority, and we will take the learnings from this incident to strengthen our processes going forward. If you have any questions, please reach out to our Support team. You can do so by visiting help.optimizely.com, logging in, and selecting the ‘File a Support Ticket’ option in the top right of the page.
The Optimizely Team