0 Likes

Keep users bucketed in variation even when experiment is 'paused'

Status: New
by Jeroen ‎06-22-2015 June 22, 2015

When I am correct: user are unbucketed from a variation of an experiment when experiment is 'paused'.

Promblem is when you restart the experiment they can get into a different variant then the one they were originally in, making the experiment in-valid.

So, my proposal is to keep users bucketed in the same variant even when expermint is paused/restarted again, and un-bucket users once the experiment gets archived.

(see info in ticket Request #99731).

Level 2

Comments
by JDahlinANF
‎06-22-2015 June 22, 2015

@JeroenYou sure about that?

 

I have an experiment I am running in our test environments (2775891173).

This is in the optimizelyBuckets cookie as:

2775891173	"2790230825"

Which shows that I am bucketed into variation 2790230825 which can be easily viewed in the console using this code (which relies on the jQuery cookie plugin), you can instead simply view the cookie if you wanted to:

var thisExperiment = 2775891173;
$.cookie('optimizelyBuckets').substring($.cookie('optimizelyBuckets').indexOf(thisExperiment)+13,$.cookie('optimizelyBuckets').indexOf(thisExperiment)+23);

After pausing the variation (wait 2 minutes), refreshing the page and verifying the experiment is not active anymore, when I look in the optimizelyBuckets cookie I see that it still contains the experiment that is paused, and it still contains the same variation ID.

 

 

 

 

Level 2
by Jeroen
‎06-23-2015 June 23, 2015

Thx nap0leon for the work!

 

I had a conversation about this with the helpdesk, they told me that this was not the case.

 

Did you check after turning on the experiment again? 

Perhaps it overwrites?

 

I'll try out your explanation later, Thx again.

 

Level 2
by JDahlinANF
‎06-23-2015 June 23, 2015 - edited ‎06-23-2015 June 23, 2015

@JeroenYup - I'm positive.  I have never moved from one variation to another variation by pausing (and reactivating) an experiment. 

 

Maybe the Randomicus (the Roman god of Random Numbers) has worked against me and I've randomly stayed in the same variation every single time we've paused and re-started experiments (which happens quite often - especially in our QA environment).

 

(Of course, I don't actually check my variation each and every time and I frequently clear my cookies, so I am not saying it is impossible that what they described is true, but I would bet on a misinterpretation of the question or answer rather than that statement being factually accurate.)

Level 2
by Jeroen
‎06-23-2015 June 23, 2015

haha, ok.

 

I'll check back with the helpdesk on this!

 

Thx.

Level 2
by Optimizely
‎06-23-2015 June 23, 2015 - edited ‎06-23-2015 June 23, 2015

Hi @Jeroen and @nap0leon! My name's Nils and I'm part of Optimizely's Technical Support team over here in Amsterdam.

 

What @nap0leon says is correct, thanks for your responses! What my colleague meant is that when you archive an experiment, the experiment is removed from the snippet. But the experiment-variation ID combination isn't removed from the optimizelyBucketsCookie set for the visitor. Then when the experiment is paused, archived, then unarchived and run again, the visitor will still have that experiment-variation ID combination in their optimizelyBucketsCookie, and will see the same variation again that they've seen before.

 

This is the same in the app, but with the app-variation of a cookie. So in the app, the visitor will also see the same variation again for that experiment.

 

I hope this helps Smiley Happy Please let me know if you have any further questions about this!

 

Optimizely