Ask the Expert: Tom Fuertes, CTO of CROmetrics

[#1]

The live Q&A portion of this AMA is closed. However, I would encourage you to read through all the amazing content that has been discussed here. 

Unsure about how to set up a complex experiment? Still learning the ins and outs of Optimizely X? Can’t figure out the solution to a problem? Now's your chance to have all of your technical Optimizely questions answered by one of Optimizely's top partners, CROmetrics.

Respond to this post with questions you may have, bounce ideas around, and get to know the expert!

Hi Optiverse! I’m Tom Fuertes, CTO of CROmetrics, where I specialize in solving complex problems with elegant solutions, data-driven experimentation and analytics, and crafting new connections between client optimization and company success.

My passion for numbers stems from my professional poker-playing days which directly influenced my skills in conversion optimization. I’d love to share what I’ve learned over the years and help you and your company/clients achieve success.

Let’s tackle your technical Optimizely questions together to find a solution in my Optimizely AMA: ASK ME ANYTHING!

You can ask questions until noon CST on Friday, March 24 at which point this session will be closed. Check out the post here for guidelines and additional information.

[#2]

@tomfuertes has been an amazing Community member answering more than five questions in the last week alone!

Tom: many clients I want to work with want to know when they should run a full-factorial multivariate test (MVT) vs running back-to-back a/b/n tests. Do you have any best practices you follow when deciding which type of test to run?

Thanks!

[#3]

@jason-gsel: "many clients I want to work with want to know when they should run a full-factorial multivariate test (MVT) vs running back-to-back a/b/n tests. Do you have any best practices you follow when deciding which type of test to run?"

Great question! We prioritize based on a lot of factors. Our decision tree is roughly as follows, but the end result is that we run very few MVT's and almost always run A/B/n's.

1. How long will 3 vs 4 variations take to reach significance? We use Optimizely's Sample Size Calculator to get a good feel for how long adding an additional variation will take to the length of the test. If you run an A/B/C (3 variations) test its usually much faster than a AB/AC MVT (4 variations).
   
2. Do the variations have the same KPI? If so then an A/B/n make more sense so you can learn the most about how each variation affects the KPI.
3. Do the likelihood of interactions heavily overlap? If not it might make sense to run then as an A/B/n instead of an MVT so you get more information on how each one affects the main KPI more quickly.
4. Do you have a solid hypothesis backed by data? If not MVT's are a great way to learn directionally how elements on your site overlap. They're great for "exploratory testing". 

Hope that helps!

[#4]

Hi Tom, it's been requested of me to try to create a conditional upsell graphic that only appears in the cart when certain skus are present. In this case it's a short list of 4 or 5, so I'm thinking it would be a "search for array". I'm not super-javascript-savvy, but I can get around. What I'm trying to figure out is how to set up the conditional activation in this case (and specifically, how do I write the code). There are elements in the page source (and we use Tealium, so I have data in the utag I can poach) that would be consistent enough to look for the skus, but I'm not sure of the best approach here. For reference, this is www.yankeecandle.com. It seems like this would be a great tool to have for other instances as well, as we're kind of limited in our upsell abilities right now on our platform. Any ideas are greatly appreciated.

[#5]

@mstrong - great question! The dataLayer definitely makes it easier. You'll need custom code and conditional activation to accomplish this. Something like below should work: 

function (activate) {

  // populate an array with sku's to look for 
  var skus = '12345,67890'.split(',');
  
  // poll for utag_data and activate if sku in cart
  (function poll() {
    var utag_data = window.utag_data;
    if (!utag_data) return setTimeout(poll, 50);
    var product_sku = utag_data.product_sku || [];
    var shouldActivate = false;
    product_sku.forEach(function (sku) {
      if (skus.indexOf(sku)>=0) {
        shouldActivate = true;
      }
    });
    if (shouldActivate) activate();
  })();
  
}

Here's the docs on conditional activation: https://help.optimizely.com/Build_Campaigns_and_Experiments/Activation_Mode%3A_Activating_an_experim...

[#6]

Tom, thank you so much - that's working right out of the gate!  Some people are going to flip around here when I show them!

Mike

[#7]

Anytime @mstrong! We have hourly engagements at CROmetrics if you need any more help. I'll also be here till Friday -Tom

[#8]

Hey Tom,

Thanks for doing this. One thing I've always wondered: Would you rather fight 100 duck-sized horses or 1 horse-sized duck?

Thanks,

Tom