Rich-text Reply

How to avoid polluting production statistics from developer testing (on free version)

abbood 09-08-15

How to avoid polluting production statistics from developer testing (on free version)

I was wondering what is the best way to avoid polluting the production (live) stats by the stats of our testers and developers?

 

Typically the way we solve this problem is simply by creating two different apps: one for dev and one for prod (we do that with hockeyapp for example).. with Optimizely.. the free version only supports one iOS app.. so the above option is no longer available.. 

 

any idea how to get around that?

Level 1

MartijnSch 09-08-15
 

Re: How to avoid polluting production statistics from developer testing (on free version)

Don't load the SDK in the development environment at all? Or make sure that you can exclude their IP address?
abbood 09-08-15
 

Re: How to avoid polluting production statistics from developer testing (on free version)

i don't think that's a good idea.. i *need* my devs to be able to test adding experiments on the development environment (the preview feature offered by optizerse not withstanding). 

 

I guess optimizely won't charge me for the test app since it won't make it to a live audience anways..

 

this is what i have in my code:

 

 

#ifdef RELEASE
    DDLogInfo(@"initializing mixpanel for prod env");
    [Optimizely startOptimizelyWithAPIToken:@"%prod_token%" launchOptions:launchOptions];
#else
    DDLogInfo(@"initializing mixpanel for dev env");
    [Optimizely startOptimizelyWithAPIToken:@"%dev_token%" launchOptions:launchOptions];
#endif

 

Level 1
Pam 09-14-15
 

Re: How to avoid polluting production statistics from developer testing (on free version)

Hi @abbood, great question!  What you have suggested is definitely one option.

 

Other options include:

  • Have your devs create experiments that are always targeted toward a custom tag or specific app version (i.e. never targeted toward live users).  Once you're ready to run this experiment on a live audience, you can duplicate the experiment and run it toward the audience you want to run the test on.
  • Create the experiments, then you can reset results once you're ready to run the test live.  A screenshot is attached.
reset_results.png
Pam
Optimizely