Respond to this post with questions you may have, bounce ideas around, and get to know the expert!
Hi Optiverse! I’m Tom Fuertes, CTO of CROmetrics, where I specialize in solving complex problems with elegant solutions, data-driven experimentation and analytics, and crafting new connections between client optimization and company success.
You can ask questions until noon CST on Friday, March 24 at which point this session will be closed. Check out the post here for guidelines and additional information.
@jason-gsel: "many clients I want to work with want to know when they should run a full-factorial multivariate test (MVT) vs running back-to-back a/b/n tests. Do you have any best practices you follow when deciding which type of test to run?"
Great question! We prioritize based on a lot of factors. Our decision tree is roughly as follows, but the end result is that we run very few MVT's and almost always run A/B/n's.
- How long will 3 vs 4 variations take to reach significance? We use Optimizely's Sample Size Calculator to get a good feel for how long adding an additional variation will take to the length of the test. If you run an A/B/C (3 variations) test its usually much faster than a AB/AC MVT (4 variations).
- Do the variations have the same KPI? If so then an A/B/n make more sense so you can learn the most about how each variation affects the KPI.
- Do the likelihood of interactions heavily overlap? If not it might make sense to run then as an A/B/n instead of an MVT so you get more information on how each one affects the main KPI more quickly.
- Do you have a solid hypothesis backed by data? If not MVT's are a great way to learn directionally how elements on your site overlap. They're great for "exploratory testing".
Hope that helps!
@mstrong - great question! The dataLayer definitely makes it easier. You'll need custom code and conditional activation to accomplish this. Something like below should work:
function (activate) {
// populate an array with sku's to look for
var skus = '12345,67890'.split(',');
// poll for utag_data and activate if sku in cart
(function poll() {
var utag_data = window.utag_data;
if (!utag_data) return setTimeout(poll, 50);
var product_sku = utag_data.product_sku || [];
var shouldActivate = false;
product_sku.forEach(function (sku) {
if (skus.indexOf(sku)>=0) {
shouldActivate = true;
}
});
if (shouldActivate) activate();
})();
}Here's the docs on conditional activation: https://help.optimizely.com/Build_Campaigns_and_Experiments/Activation_Mode%3A_Activating_an_experim...
-Tom
