Experiment Locked on iOS
I've recently added the iOS SDK to our project, and I was able to run my first test successfully. I'm now trying to add a second test and when I inspect the OptimizelyExperimentData, it has a state of OptimizelyExperimentDataStateDeactivated, and a Locked value of YES, as you can see :
These tests may be unique in that I only use a single boolean Optimizely variable to determine UI behavior clientside. I do this because these tests are somewhat complex, and have logic differences, apart from just the UI changing. Here, you can see I use the optimizely variable inPasswordCollectionTest to set a different nib, if I've set this value to YES in the Optimizely web interface. If you can't read that snippet, here's a higher rez version http://cl.ly/image/1N193l3R3O0j
Here are the two variants for my first test "Welcome Popup Copy". Again, this test is working fine.
And for my new test "Password Collection", I've done the same thing, simply turned on a boolean in Variation #1...
My inclination is that the Password Collection (problematic) test is locked because I'm (unwantingly) setting the same variable in both tests (which I know is bad). Because all optimizely variables are global, they all show up in the variables panel for any test. For example, inWelcomePopupCopyTest is set to NO in both variants of my second test "Password Collection", but I don't care about these variables here and I don't want to even see them, much less modify them.
My question is then, if I don't actively modify a boolean from its default value, in the Optimizely web interface, does a test still think that I'm tyring to modify it?
Why else could my second experiment be locked? I'm stumped!
@jonstoked Thanks for your question! I took a look at your 2 experiments, and the experiments look like they are set up as expected. It looks like you might have modified your "Password Collection" test to use a code block instead of a live variable, which works! However, your live variables test should have worked as well. As you mentioned, since you did not modify "inWelcomePopupCopyTest" in your "Password Collection" test, I would not expect the "Password Collection" test to be locked.
Even though all variables show up in Optimizely's experiment editor, as long as you don't modify the value of the variable, the variable will not be marked as modified. In addition to checking the data object, which you have done, you can also verify that a variable has not been modified by looking at your project's JSON file: http://cdn.optimizely.com/json/ios/1.0/<project_id>.json.
We are currently working on a new user experience, which should help resolve some of the confusion around which experiements modify which variables. Feel free to reach out if you have additional questions on this!
Thanks fro your help Pam! Yeah, I switched to using Optimely's blocks, and I simply set my local _inPasswordCollectionTest bool to yes inside a block that I only call in the test case (varation #1). This worsks swimmingly, with no conflicts.
I'm not sure what I was doing wrong with the bools. I even tried recreating the tests from scratch again with bools, and a third time with strings (set to "YES") if you are in the test case, and in all those cases, one of my two tests would always be locked.
Yeah, it would be nice if there was an explicit list of changes in each variation, so you could review and delete any accidental changes you may have made. But other than this, Optimizely is working great for us! Way better than Mixpanel's A/B testing tools.
@jonstoked Thanks for the product feedback! Glad to hear things are working well for you. Definitely send over any additional feedback you have as you continue using Optimizely. We want to build a product that you love using!