Rich-text Reply

Paired variants across experiments

alanho 09-21-15
Accepted Solution

Paired variants across experiments

[ Edited ]

Hi, I have two experiments running on the same host:

 

  1. Sign Up: Original & Variant A
  2. Sign In: Original & Variant A

Basically I would like to test out Variant A (which is just applying CSS color themes) for both my Sign In and Sign Up pages in a paired manner. The intent is to have the subject see either Original or variant A across both pages given that there's a possibility of jumping across the pages.

 

Can someone guide me to some documentation or steps on how to achieve this?

- Alan Ho
Level 3

JDahlinANF 09-21-15
 

Re: Paired variants across experiments

[ Edited ]

@alanho This is a pretty standard question, with a few ways to accomplish.

 

Given the lack of information in your question, I'll presume some knowledge of how your site is set up - hopefully the descriptions are close enough to allow you to proceed.  If not, please repoly with whatever additional information is required to give a more appropriate answer.

 

1- set up one experiment with URL targeting to the URLs for both the sign-up and the sign-in page.

2- in the code editor for the "B" variation, inject the appropriate CSS for each page

 

/* _optimizely_evaluate=force */
$('head').prepend('<style>#signup .someclass {somestyle:value !important;}</style>');

$('head').prepend('<style>#signin .someclass {somestyle:value !important;}</style>');

/* _optimizely_evaluate=safe */

This presumes that your CSS can be written to apply to only the signup and signin pages.

If the CSS needs to be applied based on URL or something else, you can put the above inside an if-then type evaluation.

alanho 09-21-15
 

Re: Paired variants across experiments

Thanks @nap0leon,

 

I'm thinking... by setting up a single experiment, would I be able to easily parse the results between the 2 pages?

 

In other words, my objectives are:

 

  1. Know how conversion goals are impacted for Sign Ins before and after change
  2. Know how conversion goals areimpacted for Sign Ups before and after change

 

I just looked up Optimizely UI and see that there's an ability to add Visitor Dimension. Would segmenting my results with this dimension be the way to find out 1 & 2 above?

 

Thank you for your patience, I'm still a newbie at Optimizely.

 

- Alan Ho
Level 3
JDahlinANF 09-22-15
 

Re: Paired variants across experiments

@alanho

 

By using one experiment for both pages, users will either be in the A or the B group on both pages.

I would set up separate goals that measure the "submit" button on each page and add both goals to the experiment.

 

In your dashboard, each experiment would have 2 goals listed.  You can see which experiment generates more of each "Submit".  Since each submit type is unique to the page where it happens, you will know what the lift is for each submit.

 

If you are on the Enterprise Plan (vs the Free plan), you can set this up as a multi-page experiment in which case your Sign In page will be listed separately from the Sign Up page.  But the number of users in each will still include users who saw the other page.  So nothing is really gained by doing this.

 

If you truely want to break out users, you actually need 3 segments - "Sign In Only", "Sign Up Only", and "Both".  To do this you could create a Custom Dimension named like "Account Funnel".  Since users can only have one value at a time, you could set it up like this...

1- on the Sign In page: set value to "Sign In" or "Both"

2- on the Sign Up page: set value to "Sign Up" or "Both"

 

To be able to set the value to "Both", you'll need a mechanism of tracking which pages the user may have seen before.  To keep this post from being overly long, I'll make a spearate reply walking through how to set this up.

JDahlinANF 09-22-15
 

Re: Paired variants across experiments

[ Edited ]

@alanho

 

To set up "SignIn" vs "SignUp" vs "Both" segmentation.

 

1- Set up a Custom Audience named "Account Funnel".  For the "Dimension API Name" field, enter the value "account_funnel".  Now we have a place to store the values.

 

2- Lets start with some code that determines if a user is currently on the Sign Up or the Sign In page.  Of course, you would need to modify this.  In this example, I look to see if the URL contains "signin" or "signup".  This could be changed to any javascript definition that works... perhaps a body id value, or similar.  When this code runs, "currAcctPage" will have a value of "SignIn", "SignUp", or "" (empty).

 

var currAcctPage = (window.location.toString().indexOf('signin') > 0) ? "SignIn" : (window.location.toString().indexOf('signup') > 0) ? "SignUp" : "";

 

3- if the user is on the SignIn or SignUp page, we will set a value into localStorage so that we can remember that the user saw this page in case they switch to the other page.

 

var currAcctPage = (window.location.toString().indexOf('signin') > 0) ? "SignIn" : (window.location.toString().indexOf('signup') > 0) ? "SignUp" : "";
if (currAcctPage == "SignUp") { localStorage['acctFunnel'] = "SignUp"; } if (currAcctPage == "SignIn") { localStorage['acctFunnel'] = "SignIn"; }

 

3- But - what if the user already saw the other page?  Let's add some code that checks localStorage and if the user already saw the other page, let's set that value to "Both" (we also need to check to see if it already has the value "Both").

var acctFunnel = localStorage['acctFunnel'] || "";
var currAcctPage = (window.location.toString().indexOf('signin') > 0) ? "SignIn" : (window.location.toString().indexOf('signup') > 0) ? "SignUp" : ""; if (currAcctPage == "SignUp") { if (acctFunnel == "Both" || acctFunnel == "SignIn") { localStorage['acctFunnel'] = "Both"; } else { localStorage['acctFunnel'] = "SignUp"; }} if (currAcctPage == "SignIn") { if (acctFunnel == "Both" || acctFunnel == "SignUp") { localStorage['acctFunnel'] = "Both"; } else { localStorage['acctFunnel'] = "SignIn"; }}

4- Now that we are recording whether a user has only seen the SignIn or SignUp pages, or if a user has seen "Both", let's push that value to optimizely.  To do this, we'll fetch the value from localStorage and push it to Optimizely.

var acctFunnel = localStorage['acctFunnel'] || "";
var currAcctPage = (window.location.toString().indexOf('signin') > 0) ? "SignIn" : (window.location.toString().indexOf('signup') > 0) ? "SignUp" : "";

if (currAcctPage == "SignUp") {
  if (acctFunnel == "Both" || acctFunnel == "SignIn") {
    localStorage['acctFunnel'] = "Both";
  } else {
    localStorage['acctFunnel'] = "SignUp";
  }
}

if (currAcctPage == "SignIn") {
  if (acctFunnel == "Both" || acctFunnel == "SignUp") {
    localStorage['acctFunnel'] = "Both";
  } else {
    localStorage['acctFunnel'] = "SignIn";
  }
}

acctFunnel = localStorage['acctFunnel'] || "";

optimizely.push(['addToSegment', 'account_funnel', acctFunnel]);
window.optimizely.push(['trackEvent', "sendSegmentData"]);

Now your results dashboard's "Segment" drop-down will contain a breakdown for "Account Funnel" which records values for "SignIn", "SignUp", and "Both".

 

Now that I am done writing this, I suggest you change the values to "SignIn Only" and "SignUp Only" - wouldn't want someone to pull the report for "SignIn" and accidentally think they are looking at all users who saw the Signin page.

alanho 09-22-15
 

Re: Paired variants across experiments

Thanks @nap0leon.

This works for me. Smiley Very Happy

- Alan Ho
Level 3
JDahlinANF 09-23-15
 

Re: Paired variants across experiments

@alanho BTW - I reached out to my contacts at Optimizely to look this over to make sure I'm not sending you the long way around to get to something they may have a more direct answer for.  I don't do a lot of segmenting with their dashboards (we track everything in SiteCatalyst), so there may be nuances that I'm not familiar with.  If you don't hear anything by the end of the week, I'd presume my answer is sufficient.

Re: Paired variants across experiments

[ Edited ]

Hi @alanho and @nap0leon,

 

I'm Becca Bruggman, a Solutions Architect over here at Optimizely. Thanks @nap0leon for a very comprehensive response to this question and reaching out to our team! I reviewed the above solution and wanted to chime in about this issue to note that while @nap0leon's solution would work great, given that you want to see this level of granularity around your user's behavior, that there is a way to do this using the optimizely data object versus having to set additional information in local storage. I'll outline the steps below:

 

1. Set up the experiment with the url targeting to both your Sign-up and Sign-in pages with the appropriate visual changes living in the variation code.

 

2. Create a custom dimension within your project home page then note the dimension id and dimension name. My dimension name in the example below is login_test.

 

3. Go back into your experiment and within the experiment Javascript include the following code:

 

var sign_in_page = SIGN_IN_PAGE_STRING_VALUE_TO_CHECK;
var sign_up_page = SIGN_UP_PAGE_STRING_VALUE_TO_CHECK;
var dimension_id = DIMENSION_ID;
if(window.location.toString().indexOf(sign_in_page) > 0 && !(dimension_id in optimizely.data.visitor.dimensions)) { optimizely.push(['setDimensionValue', 'login_test', sign_in_page]); } if(window.location.toString().indexOf(sign_up_page) > 0 && !(dimension_id in optimizely.data.visitor.dimensions)) { optimizely.push(['setDimensionValue', 'login_test', sign_up_page]); } if (optimizely.data.visitor.dimensions[dimension_id] === sign_up_page && window.location.toString().indexOf(sign_in_page) > 0) { optimizely.push(['setDimensionValue', 'login_test', 'both']); } if (optimizely.data.visitor.dimensions[dimension_id] === sign_in_page && window.location.toString().indexOf(sign_up_page) > 0) { optimizely.push(['setDimensionValue', 'login_test', 'both']); }

 

 

So this might look like:

 

 

var sign_in_page = 'sign_in';
var sign_up_page = 'sign_up';
var dimension_id = 12345678;
if(window.location.toString().indexOf(sign_in_page) > 0 && !(dimension_id in optimizely.data.visitor.dimensions)) { optimizely.push(['setDimensionValue', 'login_test', sign_in_page]); } if(window.location.toString().indexOf(sign_up_page) > 0 && !(dimension_id in optimizely.data.visitor.dimensions)) { optimizely.push(['setDimensionValue', 'login_test', sign_up_page]); } if (optimizely.data.visitor.dimensions[dimension_id] === sign_up_page && window.location.toString().indexOf(sign_in_page) > 0) { optimizely.push(['setDimensionValue', 'login_test', 'both']); } if (optimizely.data.visitor.dimensions[dimension_id] === sign_in_page && window.location.toString().indexOf(sign_up_page) > 0) { optimizely.push(['setDimensionValue', 'login_test', 'both']); }

From there you can set-up your goals specific to each page and filter your results based on these dimensions on the results page.

As an aside, if you are ok with only seeing the granularity of how folks are interacting with just the SignIn page or just the Signup page without knowledge of whether they have seen the other, then the multipage option that @nap0leon noted would work well and be a bit more straightforward in terms of a set-up. If being able to see more granularity in terms of what pages each visitor has seen is important, then the use of the above experiment JS code will work well.

 

Let me know if you have any questions on the above, does this help with getting this experiment set-up?

 

Thanks,
Becca

alanho 09-28-15
 

Re: Paired variants across experiments

Fantastic replies.
Both solutions are great.

I eventually went with Becca's for its simplicity; and the ability to segment in the report by Custom Dimensions. Thanks again for the solutions.
- Alan Ho
Level 3
JDahlinANF 09-29-15
 

Re: Paired variants across experiments

@Becca_Bruggman - Good to know!  I had forgotten that they are availabvle in the dimensions object (even though they are not stored in a cookie).