Rich-text Reply

Known Issue: Results not updating for some experiments

robinp 02-05-15

Known Issue: Results not updating for some experiments

[ Edited ]

Final update at 5:30 PM PST on 2/18/2015

Hi everyone-

My team and I have spent the last week digging into this incident. I wanted to post again with the update I promised about what happened and how we’re ensuring it won’t happen again.

 

On February 4th, a routine maintenance task triggered a service incident which caused a fraction of data on our storage servers to become unavailable. This issue specifically affected customers with experiments created in the last six months.

 

The result was that customers could not view up-to-date experiment results. Customers experienced these outages between twelve hours and several days.

 

We are sorry this happened. In the spirit of being transparent and open with our customers, we want to share the steps we’re taking to ensure this doesn’t happen again:

 

  1. Safeguards have been implemented around the maintenance task and associated tools. It is no longer possible for this class of mistakes to happen again.
  2. We’ve also made a number of improvements to our incident process to provide faster response and recovery in the future.
  3. Our emergency recovery approach was functional but slow. Going forward, we will run regular recovery drills as part of our standard operating practices. We are also making a substantial infrastructure investment in a parallel results service which will allow us to address issues like this without disrupting customers.

Thank you again for your patience as we worked through this problem.

-Erika Palmer (empdev)

 

Update at 11:00 AM PST on 2/8/2015

I am very happy to say that all affected experiments have now been fixed and have completely up-to-date information.

 

We are aware of an isolated case where a paused experiment is seeing cached information from during the outage. If you suspect you’re in that situation, a temporary workaround is to start the experiment and reload your results page (you can then re-pause the experiment). We expect to resolve this case and any similar cases on Monday.


My team is reviewing in depth what happened and will share out how we are changing our processes and technology to prevent this issue from ever happening again. Thank you for your patience as we’ve worked through this problem. Our engineering team is taking this issue very seriously. We are sorry for the impact this has had on you, our customers.

-Erika Palmer (empdev)

 

Update at 8:00 PM PST on 2/7/2015

I have no new information to share at this time. My team is continuing work on fixing the remaining affected experiments (created in September through November 2014). We are fixing all remaining experiments in a single batch. We are making good progress; this one-batch approach means it will be a bit longer before the final group of experiments is fixed and up-to-date.


I will post my next update by 11am PST tomorrow (2/8/2015) and as more information is available.

 

Update at 4:00 PM PST on 2/7/2015

No new information to share at this time. We are continuing work on fixing the remaining affected experiments (experiments created in September through November 2014). I will post my next update by 10pm PST on 2/7/2015 and as more information is available.

-Erika Palmer (empdev)

 

 

Update at 10:30 AM PST on 2/7/2015

All experiments created after 4pm PST on 11/30/2014 are completely fixed and now display up-to-date results. I’ll post my next status update by 4pm PST this afternoon and as more information is available.

-Erika Palmer (empdev) 

 

Update at 10:00 AM PST on 2/7/2015

All experiments created after 4pm PST on 1/16/2015 are completely fixed and now display up-to-date results.


To clarify the scope of this issue, no experiments created before September 2014 are affected. I’ve also updated the main post with this additional information.

-Erika Palmer (empdev)

 

 

Update at 6:30 PM PST on 2/6/2015

All experiments created after 4pm PST on 1/31/2015 are completely fixed and now display up-to-date results. 

-Erika Palmer (empdev)

 

Update at 6:00 PM PST on 2/6/2015

All experiments created after 4pm PST on 2/2/2015 are completely fixed and now display up-to-date results.

 

We are continuing to work on a resolution, but are now anticipating that we will not complete all the work to fix all 2015-created experiments by end of day today due to an unexpected issue we have now resolved. I will update this post as further groups of experiments are fixed with up-to-date results and also when we have a confirmed date when all experiments will be fixed. I’ll post my next status update as needed and by 11AM PST tomorrow.

-Erika Palmer (empdev)

 

Update at 2:45 PM PST on 2/6/2015

 

All experiments created after 4pm PST on 2/3/2015 are completely fixed and now display up-to-date results. Experiments created in 2015 will be completely fixed with up-to-date results by end of day today (PST). I will update this post as further groups of experiments are fixed with up-to-date results and also when we have a confirmed date when all experiments will are fixed.

 

We are fixing the most recently created experiments first, so the vast majority of affected experiments (about 80%) will have access to their latest results over the weekend.

 

A fix is now in place for custom date ranges for all experiments, as described in the last update. Note that this fix will not change the results available to customers who had running experiments on or after Wednesday, 1/28. These customers will be able to use custom date ranges, but the data they see will still not include their latest results.

 

You can expect our next update by 6PM PST today.

-Erika Palmer (empdev)

 

Update at 12:20 PM PST on 2/6/2015 

 

We're working on several parallel projects to get all running experiments up-to-date with their latest results as quickly as possible. Unfortunately, with where we're at right now, I can't provide a firm estimate of when this work will be complete. This is my team's only priority and we're optimistic that we will be able to provide a clearer update in the next few hours, if not sooner. When I can give a better estimate, I will.

 

We also have a fix in process for custom date ranges which will enable all experiments created on and after September 2014 to be able to view custom date ranges again. Note that this fix will not add most recent data to the results available to customers who had running experiments on or after Wednesday, 1/28. These customers will be able to use custom date ranges again, but the data they see will still not include their latest results. This fix should be available to all customers by 2:00 PM PST.

 

We understand the huge impact this is having on our customers and are working to make it right.

 

To reiterate what was said in the original post, all data for all experiments is still being collected, and all past data is intact. We know how much you rely upon this data and are working to get this fixed quickly and correctly.

-Erika Palmer (empdev)

 

 

Update at 8:30PM, PST 2/5/2015

 

All new experiments created after 8:30PM Pacific Time on February 5, 2015 now display up-to-date results and results for segments and custom date ranges as usual. We will update this thread again when we have an estimated resolution time for all other affected experiments.

 

Full history

 

Optimizely is currently experiencing an issue that prevents us from displaying the most recently collected data for certain experiments for some customers. We want to assure you that all data for all experiments is still being collected, and all past data is intact.

 

We apologize for this issue, and are actively working to resolve it as quickly as possible. We’ll update this thread with the latest information and estimated resolution time as it becomes available.

 

What will I see if my experiment is affected?

If your experiment is affected, you will see results updated through either Wednesday 1/28/2015 OR the last time you viewed your results between Wednesday 1/28 and 2AM PST 2/5/2015, whichever is most recent. When you try to view custom date ranges or segments on your results page, you may see zeroes or out-of-date results.

 

Which experiments are affected?

The only experiments affected by this issue are experiments that were created in or after September 2014 and running on or after Wednesday evening, 1/28/2015 at 11:59PM PST. Experiments created after 1/28/2015 are collecting data, but will not show results until the issue is resolved.

 

Select experiments created on or after September 2014, but no longer running, may also be unable to view results for custom date ranges or segments.

 

When will up-to-date data be available for my experiments?

Our engineering team is currently working on a fix, and we will let you know the estimated timeframe for that fix when we know more. We will provide more updates here as they become available.

Optimizely

Re: Known Issue: Results not updating for some experiments

Robinpam,

All my experiments disappeared from my dashboard!

Is this expected? My clients are getting crazy.

Fab 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Hello, we are trying Optimizely and we are surprised to have such an important bug for more than 12 hours !
Fab
Level 1
marchibbins 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Wow, glad I managed to find this. Status pages are green across the board, what a shame.

Re: Known Issue: Results not updating for some experiments

Robinpam its a huge deal to have the experiments not working since yesterday. Please give us an update of when is gonna be resolved.

Thanks!
tscala 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Hello Robinpam, I am happy to find this thread. I have had problems viewing results over the last couple days, and an experiment I started yesterday is showing zero results, zero visitors. I am still able to view all active, paused, and draft experiments in my dashboard. I will watch this thread for updates.

Level 2
lkraav 02-06-15
 

Re: Known Issue: Results not updating for some experiments

[ Edited ]
My software developer hat says it's impossible to estimate anything here. Any timeline given is a random guess at best. It's fixed when it's fixed. I have no doubt they're putting everything they have towards working on it. How could they not, this looks like a business life/death situation to me.
--
Leho, marketing & tech architect | G+: lkooglizmus@gmail.com
Level 4
Kamel 02-06-15
 

Re: Known Issue: Results not updating for some experiments

I hope that they will make a commercial gesture in favor of customer...
http://www.autoescape.com
Level 1
empdev 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Hi everyone, I'm a Product Manager working on this. We are very sorry about this problem with results. My team and I are working very hard on fixing it. We plan to have another update later this morning PST.
Optimizely
lerchmo 02-06-15
 

Re: Known Issue: Results not updating for some experiments

This has been an extraordinarily long downtime. It's shocking that such a good product can go down for this amount of time. Hopefully some processes are improved to avoid this great tool becoming too unreliable to trust. (this isn't the first extended down time we have endured)
Level 1
AR 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Hi there:

Would love to see an update on this asap.

Thanks,
-Aditi
AR
Level 1

Re: Known Issue: Results not updating for some experiments

Hi there empdev, would love a check-in on status!

empdev 02-06-15
 

Re: Known Issue: Results not updating for some experiments

To everyone in the comments area, I've posted an update on our status above. I'll post again as soon as we have more information to share.
Optimizely
tscala 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Thank you for the details and continuing communication , empdev!
Level 2
Adomatica 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Possible to get an update?
Amanda 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Hi everyone, @empdev has updated the intial post with some additional information. We will continue to post the current status in this thread. Thanks. 

Optimizely
jdeb901 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Can I suggest that you post progress or lack of it on a set, regular basis. Say every 2 hours. Even if it is a brief "no progress yet".

Then at least people will know when to log in and check to see what is happening. Some of us use this in our work and wasting further time trying to find out what is happening is just a further waste of time.
Level 2
empdev 02-06-15
 

Re: Known Issue: Results not updating for some experiments

Hi everyone, I've posted a new update in the main post. I'll post another update by 6pm PST tonight.
Optimizely

Re: Known Issue: Results not updating for some experiments

@empdev I see the data! Thanks for the fix - it came just in time

empdev 02-07-15
 

Re: Known Issue: Results not updating for some experiments

Hi everyone, I've posted a new update in the main post. I'll post another update by 11am tomorrow morning and as needed tonight and this weekend.
Optimizely
shayda 02-07-15
 

Re: Known Issue: Results not updating for some experiments

HI Erica,

 

This is really concerning - especially because we have no insight into what is actually happening and now it looks like the data in our reports are jumbled and highly innaccurate. Looking forward to your next update. 

Level 1
empdev 02-07-15
 

Re: Known Issue: Results not updating for some experiments

To everyone in the comments area, I've posted two updates on our status in the post above. All experiments created after 4pm PST on 11/30/2014 are completely fixed and now display up-to-date results.
Optimizely
taylor4484 02-07-15
 

Re: Known Issue: Results not updating for some experiments

I've got an experiment that was created on 1/31 but it's still missing data. The experiment has been running weekly 9-4 M-F and I only have data from Wednesday - Friday. What's up here? I'm concerned about that testing data since your updates say it should be fixed.
empdev 02-07-15
 

Re: Known Issue: Results not updating for some experiments

I wanted to post a quick comment to acknowledge the people who have mentioned specific issues with their experiments/accounts. My team and I are looking into your specific cases.
Optimizely
empdev 02-08-15
 

Re: Known Issue: Results not updating for some experiments

To everyone in the comments area, I've posted an update above. All experiments affected by this issue are now completely fixed and have up-to-date results.
Optimizely
tscala 02-09-15
 

Re: Known Issue: Results not updating for some experiments

Erika Palmer, thank you for the messages and all your effort over the weekend. The missing results data from new experiments and from other experiments with a custom date range appears restored. Referencing your most recent update to the first post in the thread, I look forward to hearing from you about changing our processes and technology and the cause of this issue.

 

- Thomas Scala

Level 2
Peter 02-09-15
 

Re: Known Issue: Results not updating for some experiments

I had an experiment running last week that doesn’t seem to be running anymore and there is no historical data to show. Currently all of my experiments are in "Draft" status. Might this have been caused by the same problem or something else? Is there anything I can expect to do apart from just starting it again and waiting longer for results to come in?

Level 2
empdev 02-09-15
 

Re: Known Issue: Results not updating for some experiments

Peter, I just wanted to let you know that we are looking into this. On first glance, I don't think this is related to the incident described in this post. I'm going to have the team that works on the Dashboard and Results pages check this out.
Optimizely
Mavericks 02-10-15
 

Re: Known Issue: Results not updating for some experiments

[ Edited ]

We've experienced the same issue as Peter

Mavericks 02-10-15
 

Re: Known Issue: Results not updating for some experiments

Erica,

 

thanks for your updates on this matter .

 

we also seem to have an issue similar to that as described by Peter ( in this thread ) our account email is westernwear@mavericks.net.au

 

Could our issue also be looked at , as we have lost all data , the experiment is now in draft , and  our varation seems to have totally disapered. .....

 

thanks for your asistance in this ..... I really appreciate any assistance you or you team may provide

 

Thanks again

 

Richard Norris

 

joako 02-11-15
 

Re: Known Issue: Results not updating for some experiments

Hello,

I started 3 experiments on the 9th of Feb and I did not get any results since then.

Is it possible that this issue is still affecting me? Or what should I check?

Thank you very much for your support.

Have a nice day!
Level 2
kirizawa 02-11-15
 

Re: Known Issue: Results not updating for some experiments

Does this also impact the significance figure being 0% across all data and excperiments?
Level 2
Amanda 02-12-15
 

Re: Known Issue: Results not updating for some experiments

@Peter , @kirizawa , and @Mavericks -- Your scenarios are actually slightly different than the issue mentioned in this thread. I am going to open a support ticket on each of your behalf to ensure that it gets resolved. Thank you for your patience and understanding. 

Optimizely
empdev 02-19-15
 

Re: Known Issue: Results not updating for some experiments

To everyone in the comments area, I've posted an update above. My team and I spent the last week digging into this incident and my update outlines the steps we are taking to prevent this issue from happening again.
Optimizely
lu 02-10-17
 

Re: Known Issue: Results not updating for some experiments

It's 2017 now, this bug still exists and the most bad thing is our company just buy the license. Can optimizely solve this problem? This is really bad. We make a experiment and wait there for more than 1 hour like a idiot. Really bad bad bug

lu
Level 1