Rich-text Reply

How to Create / Measure Multi-Page, Multi-Goal Experiment?

GabrielG 01-17-17

How to Create / Measure Multi-Page, Multi-Goal Experiment?

We'd like to run a test along the following lines, but I don't know how to create such a test in Optimizely and would like the community's help in that regard, please.

 

The idea is to test new product detail page templates, for a lead generation product catalogue site. The existing pages don't have calls to action to get a quote, just calls to action to download collateral, and the same sitewide link to the contact page that the new templates also have. 

 

What I have in mind is to measure quote requests as one type of conversion, as well as contact us emails as a second type. The idea being that perhaps the new type may get more "quote request" leads, but fewer people using the "contact us" to convert, so we want to measure both and get the full picture. However I don't know how to set this up in Optimizely, since it involves multiple pages and multiple goals. 

 

Can someone explain to me how to set up and accurately measure a test as described above, please?

 

In addition, to make matters more complex, the new templates have space above the fold for a product description. These are missing from most products, and junk for many others. Should I offer to write them for the client? Should I test with the existing, problematic data they have? Doesn't that risk being a confounding variable in the test, i.e. making the new templates perform worse because they are lacking information in a somewhat obvious manner? I'm tempted to test regardless, because I think the new templates are a step up from the old and that the noticeable calls to action will get them more leads, but admit this makes me nervous. Past requests to the client to fill in the info have been unsuccessful (the client = senior staff who don't feel it's worth their time/don't know how do it). 

Level 2

Highlighted

Re: How to Create / Measure Multi-Page, Multi-Goal Experiment?

So basically you want to run a test with multi-page test
1- First Go to your Dashboard and the create a test
2-Select a name for the test and the URL of the experiment
3-At this point you have 3 Options (AB, Multivariate, and Multi-page) in your case I would recommend the Multi-page test
https://help.optimizely.com/Build_Campaigns_and_Experiments/Experiment_Types%3A_AB,_Multivariate,_an...
4- Select the traffic Allocation (% traffic for the experiment and % to use in every variant)
https://help.optimizely.com/Target_Your_Visitors/Traffic_allocation_and_distribution
5- You need to define your goals
https://help.optimizely.com/Measure_success%3A_Track_visitor_behaviors/Goals%3A_Measure_success_in_O...

So at this, point I will recomend you start to play with dashboard, get familiar with the settings and terms
https://www.optimizely.com/optimization-glossary/

make some test just for learn, trust me Optimizely
is a solid tool but is a little bit complex if you do not know what you are doing
If you need any help you are welcome in the community
SEO and CRO Specialist

Re: How to Create / Measure Multi-Page, Multi-Goal Experiment?

GabrielG 01-22-17
 

Re: How to Create / Measure Multi-Page, Multi-Goal Experiment?

Hi Roman,

Thanks but that doesn't really answer my question. Though it seems I may need URL targeting plus some help to test a template whose content is dynamically populated from a database. Can I do that on the free plan? Here's what I tried emailing Optimizely before finding at no longer works


I'm trying to figure out if I can use your tool on the free plan to run the following experiment. We have a new product detail page template with a quote request button. The old template has no such button; visitors had to find the contact page link and go there to convert. Can I create a test where we see if we get more leads in total (quote request plus contact page) with the new pages vs the old, on Optimizely free?

My idea is to use URL targeting, but then it's unclear from the documentation how I would implement a change to a template that is dynamically populated with database content, as opposed to a static element. Your example of a site wide change was altering the nav bar, which can even be done in the visual editor, so I'm not aware if a change to a dynamic element is possible. Can you help please? I'm happy to pay for support if possible/relevan
Level 2
JasonDahlin 01-23-17
 

Re: How to Create / Measure Multi-Page, Multi-Goal Experiment?

"What I have in mind is to measure quote requests as one type of conversion, as well as contact us emails as a second type. The idea being that perhaps the new type may get more "quote request" leads, but fewer people using the "contact us" to convert, so we want to measure both and get the full picture. However I don't know how to set this up in Optimizely, since it involves multiple pages and multiple goals. "

 

You can create a goal that counts how often either of the forms are submitted or how often either of the thank-you pages are loaded, or similar.  In the case of counting form submissions, you would create a "click goal" and the jQuery selector for the goal would be whatever you need it to be to capture both form's submit buttons, perhaps something like this which would record whenever the "submit" button is clicked on the "contact us" or a the "request quote" forms (your CSS will surely be different, this is just an example):

.contact-us button[name=submit], .request-quote button[name=submit]

You could also set up click-goals to measure each of those form submissions as separate goals too!  That way you can see what the ratio of form usage is between the two variations and overall.

 

As for how to set this up.... For the experiment itself, this sounds like a redirect experiment (sending users from mysite.com/something/page_name_here to mysite.com/somethingElse/page_name_here)?  Since you are doing several pages instead of a single page, you'll probably need to use custom code to perform the redirect.

 

As for the database not having information for each section of the new page... you'll definitely want to do something so that the pages don't look like garbage when entire sections are emty.  Perhaps some code that looks for the text of each section and if there is no text present it injects some CSS that hides that section (or remove that section from the page, etc.).  What you do for each of the sections that may be empty will depend on how responsive the page is and you deciding if it is better to leave it blank or to remove it (if a section is blank but the space needs to exist for formatting purposes, maybe just remove the headline in the section so that it looks like that blankness is intentional).  Lots of possibilities here, but it depends on your page's design and how well it accomodates different sections missing.

--Jason Dahlin
Analytics and Testing Guru Smiley Happy
Experimentation Hero
GabrielG 01-24-17
 

Re: How to Create / Measure Multi-Page, Multi-Goal Experiment?

Hi Jason,

You're right on the thankyou pages and that would be the simplest solution.

Regarding the question of dynamic content, I see what you're suggesting as far as the redirects ... that sounds plausible.

Can you help me figure out the details of this, please? The product detail pages all end with -P[1-3 digits].html . So I want to use URL targeting and have a regex that matches e.g. -P19.html and -P4.html and -P831.html .
It seems this would cause difficulty if I also want to do this with a redirect experiment. The problem is that you're told with URL targeting that you should NOT target a pattern that matches both the original and alternative pages (because you'd cause an infinite redirect loop). If I wanted the alternative pages to be at -P[1-3 digits].html?template=b, then my regex targeting the string -P[1-3 digits].html would also match the variations.
Actually, I think I just figured out the solution to that. In URL targeting, I should use a regex that matches the full URL e.g. domain.com/product-name-P[digits].html. And for the redirect, rather than a parameter, I should have the visitors send to a new path, e.g. domain.com/template-b/product-name-P[digits].html . Does that make sense? I'd love to have a third party confirm that this can work or tell me why it can't.
Level 2
JasonDahlin 01-25-17
 

Re: How to Create / Measure Multi-Page, Multi-Goal Experiment?

For URL targeting, set the "include" to be the pattern of the pages as you described; for the "exclude", remove anything with "template=b"
--Jason Dahlin
Analytics and Testing Guru Smiley Happy
Experimentation Hero