Rich-text Reply

Users still seeing Original even after Variation was set to 100%

SavinaV 04-18-16

Users still seeing Original even after Variation was set to 100%

Hi!

 

I am experiencing the following scenario: I have a test that has been running for some time. Initially it was a 50/50 A/B test between an Original and a Variation. After a while it became clear that the Variation performed better, so in order to start taking advantage of the lift while the developers are working on the change, I changed the experiment to show the Variation for 100% of traffic and the Original for 0%. However, some users have been seeing the Original in the timeframe after the move to 100% Variation (consistently until the present moment). This is based on both the Optimizely results dashboard and examples from actual users.

 

What's going on and is there a way to prevent it?

 

Thank you,


Savina

Level 1

JasonDahlin 04-18-16
 

Re: Users still seeing Original even after Variation was set to 100%

@SavinaV - experiments persist.

 

Once a user is placed into the "A" group, they will stay in the A group forever.

Now that you have it set to 100% B group, all *new* users will join the B group.

 

To accomplish what you want to accomplish, you should clone your experiment and set the B group to 100% in the clone.  Then pause the original and start the clone.  In this way, all users will now see the B experience.

--Jason Dahlin
Analytics and Testing Guru Smiley Happy

Re: Users still seeing Original even after Variation was set to 100%

It would be great if there is a way during testing of an experiment to reset that bucketing. Before we run experiments we want to make sure they work completely. At the moment I use InPrivate browsing of the various browsers but I don't know whether there are any side effects.
Andreas