What happens if the relationship between tested variables is non linear?
what happens if the relationship between tested variables is non linear, which frequently is the case when testing many many variables. will optimizely just never reach a conclusive result?
Thanks for your question. I can come up with two main cases for non-linearity in A/B Testing, and I’ll try to address them both in this response.
The first one comes from interaction effects between multiple experiments. As of right now, Optimizely does not explicitly try to estimate correlations between experiments, or any other interactions. We do conduct our randomization in a such a way that any visitor falling into one variation or baseline has an equal chance of being in any variation or baseline in another concurrently running experiment.
What this means is that we report an estimate of improvement that is averaged over the effects of any other experiment you may have currently running. While the magnitude of the improvement may be impacted, it is highly unlikely that such interaction effects would cause a test to reverse it’s call, from winner to loser. You can read previous posts we had on interaction effects here: https://community.optimizely.com/t5/Using-Optimizely/Conflict-from-running-multiple-experiments-at-t...
All the material there pertains equally well to Stats Engine. In conclusion, it will still call conclusive results, but these conclusions will be on the improvement averaged over the variations in other experiments.
The second case comes from a non-linear effect of a variation over time. An example of this is seasonality such as a weekend vs weekday effect. Stats Engine deals with this sort of non-linearity in a very similar way to interactions. Instead of explicitly trying to estimate it, Stats Engine calculates results on the improvement averaged over time since the A/B Test began.
This comes with one caveat. With Stats Engine a user is able to keep their test running even after a conclusive result is reached. Because of this Stats Engine can actually correct itself if a variation’s improvement changes due to exogenous variation, and change a no longer winning result to inconclusive while still keeping all the same inferential guarantees. You can read more about Stats Engine’s reset policy here: https://help.optimizely.com/hc/en-us/articles/200039895#corrections
Feel free to shoot over any further questions you may have.
Statistician at Optimizely