Redirect variations and right clicks not working
We have an experiment running and we’re trying to understand the results. In so doing, we’ve discovered that if you right-click on a link to get to the experiment page, then the redirect (to the variation) does not seem to happen. Why would this be?
I’ve tried the following:
* Bucketize my browser to ensure I’m in the variation (redirect experiment)
* Right click on a link to open the test page in a new tab, and the redirect does not fire
* Refresh the page (soft and hard) and still the page does not redirect (and no query params added)
* Go back to the page where the link was, and click on it normally (it happens to load in a new tab via target=“_blank”), and it redirects to the variation correctly
Can you please help / explain this?
Checking to make sure I have this right:
1- there is a page on your site that you are doing a redirect experiment on
2- this URL is available via link on your homepage (or similar)
3- 'left-clicking the link" takes the user into the experiment and the user is correctly redirected (or not)
4- right-clicking open in new tab" takes the user into the page that should redirect but is not redirecting.
1- Are you using Classic or X? (there may be subtle differences)
2- Are you incognito/private?
3- Are there any Audience Conditions?
4- Have you copied+pasted the URL from the right-click experience into the URL Validator to double-check that the URL does match the URL Targeting criteria?
When you do the right-click open in new tab and the page does not redirect, run this in your console and look to see if there is a reason for the experiment to not have run:
(If you are using Chrome, you can filter your console for the experiment ID and it will show you only the lines that are relevant to that experiment.)
Analytics and Testing Guru
Thanks for the advice. Yes, your understanding is correct.
1) We're using Classic (should we switch to X? Had a quick look and it looks like the way to go)
2) No, not using incognito
3) Yes, the audience is new users only
4) Yes, the URL is fine (the redirect simply adds a query parameter, so it's easy to see)
Great idea re the log, thanks. Here's the difference:
Left click to open tab:
Optimizely / Condition / Testing audiences for experiment XXXXXXXXXX: YYYYYYYYYY [time 21 +0]
Optimizely / Visitor / Got dimension (first_session) value 5701005575: true [time 23 +2]
Optimizely / Visitor / Checking if in audience YYYYYYYYYY: true [time 23 +0]
Optimizely / Condition / Testing URLs for experiment XXXXXXXXXX [time 23 +0]
Optimizely / Distributor / Going to distribute experiment "-- Name --" (XXXXXXXXXX) [time 24 +1]
Right click to open tab:
Optimizely / Condition / Testing audiences for experiment XXXXXXXXXX: YYYYYYYYYY [time 23 +1]
Optimizely / Visitor / Got dimension (first_session) value 5701005575: false [time 24 +1]
Optimizely / Visitor / Checking if in audience YYYYYYYYYY: false [time 24 +0]
Optimizely / Condition / Failed to match any audiences for experiment XXXXXXXXXX [time 24 +0]
Optimizely / Condition / Testing experiment ZZZZZZZZZZ [time 24 +0]
So for some reason the right click fails to identify the browser as in the audience?
Any ideas on how I (or you guys) can fix that? I'm worried it's reporting incorrectly. Or simply not reporting at all?
Hi @Dunc I'm sorry you haven't gotten a resolution yet. I'm escalating this to our technical support engineering team who will investigate and reply on this thread.
Thanks for folling up, someone else already contacted me - ticket number 153825. I think the consultant's name is Patricia.
This is happening due to how Optimizely defines+checks for a new visitor and how sessionStorage behaves.
These are the steps Optimizely takes to define a new visitor:
1. Check if the visitor has a optimizelyEndUserId cookie, if none is present, create one.
2. If the visitor is a new visitor, then drop a sessionStorage item so that Optimizely can continue to identify this visitor as a new visitor during this session.
When you open a new tab a new session is created. This means that the sessionStorage item is lost in the new tab because of the way sessionStorages are defined. From Mozilla's docs on sessionStorage:
"A page session lasts for as long as the browser is open and survives over page reloads and restores. Opening a page in a new tab or window will cause a new session to be initiated, which differs from how session cookies work."
However, the optimizelyEndUserId cookie remains in this new tab, so Optimizely sees the visitor as returning.
Hope this clarifies.
Thanks for the explanation, that now makes sense!
The documentation from Mozilla and other places seem to differ slightly, and I it wasn't clear to me why from Mozilla's page there should be a difference between 1) clicking on link that would open in a new tab by virtue of a target="_blank" and 2) the user right clicking on such a link to force it to open in a new tab anyway. So I wrote a quick JS test to test / prove this, and indeed you're right. I'm still not clear on the rationale...
However, this will affect our test results, as I see it. What I don't know is by how much. Because currently we measure (new users only):
1) A goal for users who open a form (currently up a great 17%)
2) Goals for later down the funnel, where they could have gone back to a category page, right clicked to open in a new tab, and then not been in the experiment any more when they complete the other goals.
Am I correct in that analysis?
What does this mean?
1) In future, shoudl we run tests without the segment "New Users" only? I suppose we could use that for simpler single-page-and-goal type tests?
2) I should probably look back over past tests, and see if there are similar ambiguous or error prone results?
A/B testing is complex. Also, the difference / missed users will not be pushed to Google Analytics, as they're not being bucketized!
Many thanks for the help. If you could offer advice as above that would be great.
"New Users" is definitely the cause here...
The short-version of what is happening is that the new tab is actually a new window. Your browser just happens to make this new windows look just like a new tab, because "reasons".
Given the preponderance of "Private" and "Incognito" users (some of which forbid using sessionStorage, like iOS Safari where sessionStorage exists but has a max-size of 0, so you cannot put anything into it), anything that relies on sessionStorage is going to break. Since your site uses _blank, you will need to create a work-around for this.
My suggestion is to create a Custom Audience that lets user who meet the following conditions into it:
1- the optimizelyEndUserId cookie does not exist.
2- users who have a specific session cookie (e.g., opt_experimentId, where "experimentId" is the actual ID of the Experiment that you want to use this audience for).
#1- this would be truely new users on their first page hit.
#2- when your experiment runs, you are going to create this session cookie so that when a user reaches their second page, this cookie acts as their key to getting into the experiment.
If the number of cookies on your site is an issue (or you run lots of experiments targeting new users and don't want to create a "too many cookies" issue), instead of creating a specific cookie for each experiment, you could use a single cookie that contains a deliminted list of experimentIds in it, e.g. optNewUserExperiments. The experiment code would add the value "|experimentId" and your code in #2 would look for "|experimentId".
Analytics and Testing Guru