Rich-text Reply

Interpreting inconclusive results (bayesian vs frequentist)

marie 03-03-17

Interpreting inconclusive results (bayesian vs frequentist)

I've never seen optimizely (frequentist) show a higher level of statistical significance (87% in the example) compared to bayesian CTBC (68.1%) in the example. 

Since bayesian is less accurate - only telling me what's the probability of the variation performing better than control (anywhere between 0 - 17%) - it should be showing a higher probability. no?

 

Note that in my example optimizely is saying that ~15000 visitors remaining. Does this play a role in the question above? 

 

How would you interpret/explain the 87% stat. sign.? And how would you explain why it's bigger than the CTBC shown using bayesian statistics. 

Screen Shot 2017-03-03 at 10.46.04.png
Level 2

Michal 03-07-17
 

Re: Interpreting inconclusive results (bayesian vs frequentist)

Hi Marie,

 

There are a few things that come into play here - for example, it's important to keep in mind that you have a certain Statistical significance level set as your Optimizely project setting (this is by default 95% but you can amend the value), and the lower the number there, the higher the Statistical significance percentage shown on the Results page will likely by (because the stats engine is working with a bigger margin of error).

 

Also, the Statistical significance shown in your screenshot is basically telling you what's the probability that the difference between the two variations will be within the Difference interval - which is, at the time, still pretty wide and stretching also into the minus values. The ~15.000 additional visitors is an estimate for how manny more visitors are needed before the stats engine will be able to tell with a pre-defined certainty that the Variation outperforms the Original - the Difference interval is going to be fully within positive numbers.

 

In general, you're right about the fact the Bayesian model should allow you to see statistical significance faster (also with a higher risk of having a false winner/loser) but it's not true that it will show higher statistical significance all the time. If you'd like to read more about this topic, I recommend the dedicated section on our site: https://www.optimizely.com/statistics/ and also the Knowledge Base article: https://help.optimizely.com/Analyze_Results/Stats_Engine%3A_How_Optimizely_calculates_results_to_ena....

 

Best regards,

Michal

Optimizely