Rich-text Reply

Multiple Goals / Landing Page - Looking for testing strategy

rgrossha 05-30-14

Multiple Goals / Landing Page - Looking for testing strategy

Is there a best practice for the following scenario?


We want to test conversion on a landing page that is reached both by prospects, and by returning members who are not cookied by our site (they previously logged out of our site).  They may or may not be cookied by Optimizely.


We want to eliminate the distorting effect of members who logout, return, and login, so that we can test changes to the page that affect conversion to a member.


The existing member clicks a link to login on that page.  We have established the login page that link leads to as a goal.


Conversion to a member is the other goal.


On average 30% of visitors to this page have actually previously converted into members.  But that average can vary pretty widely.


Is it as simple as making sure our sample size is large enough after discounting it 30%, and also making sure that logins from each of the variants have no insignificant variance from each other?


Thanks ahead of time.




Level 1

Kathryn 06-02-14

Re: Multiple Goals / Landing Page - Looking for testing strategy

Hi @rgrossha,


That's an interesting use case.


Something that might work in this scenario would be to set up the expeirment for only new visitors in the targeting conditions. Optimziely distinguishes between new and returning visitors as follows:


New visitors: have never been to a page containing your Optimizely snippet before

Returning visitors: have visited at least one page containing your Optimizely snippet and have thus been cookied


However, this will only work as intended if the landing page is definitely going to be the first URL which your prospects hit. if you expect them to have any visited other pages containing the snippet by the time they hit the page on which the expeirment will run, they'd already be considered a returning visitor and this scenario wouldn't work effectively.


Another option would be to use cookies to distinguish between members and prospects. Whilst I'm not familiar with your site setup, it might be possible to exclude visitors from your test if they have a persistent cookie, which only members have, for example. Or you might be able to change your setup to apply such a cookie to your members, whether or not they are logged out. You can set up a test to exclude visitors with a certain cookie in the visitor conditions segment of the targeting conditions.


If you have any follow-up questions, just let me know!

rgrossha 06-02-14

Re: Multiple Goals / Landing Page - Looking for testing strategy

Thanks.  We're trying only "new" visitors currently.  About 30% of new visitors are returning members who either have logged out, or lost their cookie somehow.


We could drop a cookie that doesn't disappear when they log out, but I think our members wouldn't like us if we did that.


My current plan is to get both a significantly significant number of new members (after discounting 30%) AND wait until the percentage of folks logging in via each variant is equal.  


Just to confirm behavior in this scenario.


1) New (never seen Optimizely snippet) visitor comes to page.

2) Gets Optimizely cookie

3) Registers (counts as a Registration conversion).

4) Returns several days later, and logins (Login Conversion).


In this scenario, we will see a conversion for each of the two goals - Registration and Login.





Level 1