Rich-text Reply

Android SDK manually activated experiment bug

fingertricks 05-10-16

Android SDK manually activated experiment bug

[ Edited ]

Using Android SDK 1.4.2 and Manually-activated Experiments I've found the following bug.

 

Steps to reproduce:

1. Manually activated experiment is configured on dashboard and Started.

2. Navigate to screen within the app that activates on the experiment in onActivityCreated(): 

Optimizely.activateManualExperiment(experimentIdentifier);

and also gets a LiveVariable<Boolean> associated with that experiment:

private static final String HYGIENE_LOGOS = "HygieneLogos";
private static LiveVariable<Boolean> enableLogosVariable = Optimizely.booleanForKey(HYGIENE_LOGOS, DEFAULT_VALUE);

 

public boolean isExperimentVariantOn() {
try {
Boolean variantEnabled = enableLogosVariable.get();
Logger.d(TAG, "Experiment (HygieneLogos) LiveVariable value is " + variantEnabled);
return variantEnabled;
} catch (ClassCastException e) {
// Ugly hack to workaround: https://github.com/optimizely/Optimizely-Android-SDK/issues/9
Logger.d(TAG, "Experiment (HygieneLogos) LiveVariable not found, defaulting to: " + DEFAULT_VALUE);
return DEFAULT_VALUE;
}
}

 

3. Lock the device.

4. Unlock the device, resuming the screen which refreshes the view and gets the LiveVariable value again.

 

Expected result: 

Same value is returned from isExperimentVariantOn() as it was originally bucketed.

 

Actual result:

The default value is always returned from isExperimentVariantOn() and the following log is seen (filtered for "Optimizely"):

 

05-10 15:31:34.847 24099-24191/com.xxx.xxx D/Optimizely Logging: [OptimizelyNetworkUtil] Executing request https://cdn.optimizely.com/json/android/kill_switch/5317080705.json
05-10 15:31:35.166 24099-24172/com.xxx.xxx D/Optimizely Logging: [amplitude_mobile] Registering
05-10 15:31:35.166 24099-24172/com.xxx.xxx D/Optimizely Logging: [localytics_mobile] Registering
05-10 15:31:35.167 24099-24172/com.xxx.xxx I/art: Rejecting re-init on previously-failed class java.lang.Class<com.optimizely.integrations.mixpanel.OptimizelyMixpanelIntegration>
05-10 15:31:35.167 24099-24172/com.xxx.xxx D/Optimizely Logging: [OptimizelyFieldsLoaderTask] Optimizely skipped scanning com.optimizely.integrations.mixpanel.OptimizelyMixpanelIntegration class to discover CodeBlocks and LiveVariables. If this is not one of your components or does not contains any CodeBlocks and LiveVariables, you can ignore this warning.
05-10 15:31:35.167 24099-24172/com.xxx.xxx D/Optimizely Logging: [google_universal_analytics_mobile] Registering
05-10 15:31:35.343 24099-24099/com.xxx.xxx E/Optimizely Logging: [OptimizelyDownloader] Cancelled download because it took longer than 500
05-10 15:31:35.432 24099-24172/com.xxx.xxx D/Optimizely Logging: [OptimizelyData] Parsed data file version 55
05-10 15:31:35.432 24099-24172/com.xxx.xxx D/OptimizelyExperimentManager: Optimizely data file loaded
05-10 15:31:35.433 24099-24205/com.xxx.xxx D/Optimizely Logging: [OptimizelyNetworkUtil] Executing request https://cdn.optimizely.com/json/android/1.4/5317080705.json
05-10 15:31:35.697 24099-24205/com.xxx.xxx D/Optimizely Logging: [OptimizelyData] Parsed data file version 55
05-10 15:31:35.698 24099-24205/com.xxx.xxx D/OptimizelyExperimentManager: Optimizely data file loaded
05-10 15:31:35.698 24099-24205/com.xxx.xxx D/Optimizely Logging: [OptimizelyData] Successfully saved data file to disk.
05-10 15:31:35.702 24099-24205/com.xxx.xxx D/Optimizely Logging: [Optimizely Logging] Optimizely SDK version 1.4.2 started
05-10 15:31:35.702 24099-24205/com.xxx.xxx D/Optimizely Logging: [Optimizely Logging] Running in Normal mode.
05-10 15:31:35.702 24099-24205/com.xxx.xxx D/OptimizelyExperimentManager: Optimizely started
05-10 15:31:35.702 24099-24205/com.xxx.xxx D/Optimizely Logging: [amplitude_mobile] Plugin cannot be activated since it has not been enabled via the data file. Consider calling Optimizely.whitelistPlugin() if you are developing a plugin.
05-10 15:31:35.702 24099-24205/com.xxx.xxx D/Optimizely Logging: [localytics_mobile] Plugin cannot be activated since it has not been enabled via the data file. Consider calling Optimizely.whitelistPlugin() if you are developing a plugin.
05-10 15:31:35.708 24099-24205/com.xxx.xxx D/Optimizely Logging: [google_universal_analytics_mobile] Plugin Activated Successfully
05-10 15:31:49.838 24099-24099/com.xxx.xxx D/Optimizely Logging: [Optimizely Logging] Attempting to manually activate experiment: 5338620580
05-10 15:31:49.839 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyBucketing] Bucketed user in variation: FSA Logos on Info Screen for experiment: FSA Logos on Info screen
05-10 15:31:49.842 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyData] Activating experiment FSA Logos on Info screen, variation = FSA Logos on Info Screen
05-10 15:31:49.862 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyData] Marked experiment: FSA Logos on Info screen as viewed
05-10 15:31:50.016 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyVariables] Returning value true for variable key HygieneLogos
05-10 15:31:50.237 24099-24191/com.xxx.xxx D/Optimizely Logging: [OptimizelyEventsManager] Sent 1 events.
05-10 15:31:50.555 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyVariables] Returning value true for variable key HygieneLogos
[Lock device]
[Unlock device]
05-10 15:32:00.739 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyData] Parsed data file version 55
05-10 15:32:04.893 24099-24099/com.xxx.xxx D/Optimizely Logging: [OptimizelyVariables] Returning default value false for variable key HygieneLogos

The last line shows that Optimizely returns the default value for the LiveVariable.

 

Workaround:

This bug can be avoided by disabling reloading experiments on foregrounding:

Optimizely.setshouldReloadExperimentsOnForegrounding(false);

Note that this is not the default behaviour. Without this workaround, screens that manually activate experiments can have unexpected results when the screen locks due to inactivity or manually locking the device.

DavidS 05-11-16
 

Re: Android SDK manually activated experiment bug

Hi fingertricks,

 

Thanks for reaching out and for reporting this issue!

I have reported this issue to our engineering team to investigate why this is happenning. I'll follow up in the ticket you have open with us.  

 

Feel more than free to reply back with any additional questions at all.

 

Best,
David

 

Optimizely