Rich-text Reply

test shows revenue increase. how do I validate it?

dfoley 06-11-14

test shows revenue increase. how do I validate it?

I have a test that shows increase in revenue by 10%. I just made the change live. How do I show that irs really working?

In ga I can view revenue by week. Should I look for a 5% increase from the week that the test was running?

What if I have a teat I just started that seems to be showing a negative revenue .. Its only 1 day in and nothing significant yet. How do I factor that in?

Also my business is seasonal. Summer sales always slowly drop..

I just need to know how to prove this increase is really happening.. Please help!
Level 2

adzeds 06-11-14
 

Re: test shows revenue increase. how do I validate it?

- Did you run your test across a specific segment of your traffic?
- Did your changes target specific pages on your website?
- Are their micro-conversions that you optimized to generate the revenue increase?
= You should be able to dig into your GA to find out if those metrics have improved.

For example, if you run an eCommerce store and you ran a test to increase 'Add to Basket' on product pages for PPC traffic, drill your analytics down to that level to see if the increase is noticable at the micro-conversion level rather than just on the 'Revenue' metric. It may be that you still have the 5% increase in 'Add To Baskets' that was the result of your test but those visitors are not completing their purchase for some reason..

It is difficult to diagnose without knowing exactly what you changed but if you put up some details I might be able to point you towards some reports in GA that would help you monitor the changes success.

I am assuming that you ran your initial test to statistical significance!
David Shaw
Level 11
dfoley 06-11-14
 

Re: test shows revenue increase. how do I validate it?

Here's what i tested.. 2 things:

 

1. made my Add to Cart button 3x the width. It looks much better as it fills up it's container. got me a 10% increase in conversions.

 

2. I used to show only 15 products per page, and let the user choose to show 60 per page if they wanted to. I set the default on 60 per page to see what would happen. It got me a 5% increase in revenue and a 2% increase in visits to the product page.

 

 

Also i killed my last expirement that was running at it was showing a 30% decrease in revenue. Only ran for 3 days, but i ran the power numbers and it had enough conversions to be a good result. 

 

So do i not run any more experiements until i can validated how much more revenue I am getting? 

 

Should i go ahead and run some more? and just make sure to take those increase or decreases into account??

 

Again how show i look for the increase? Should i look at a the 7 days with the live changes vs the 7 days before that? Should i expect to see it just in one day? 

Level 2
adzeds 06-12-14
 

Re: test shows revenue increase. how do I validate it?

OK.

So when we monitor the success of changes we have deployed we plot it against several metrics:

- Long-Run Average
- Optimizely Average
- Similar Week Average
- Last 7 Days Average

So the last 7 days average should be the 7 days since you put the change live.
The Long Run average is the average for that metric over a specified period of time, for example two months before you made the change.
The Optimizely average is the conversion rate your variation had when testing.
The similar week average is where you compare your last 7 days to a similar 7 days. So if you have put your change live on the first monday of the month, use the previous months first monday as your benchmark.

What you should now have are four averages for performance in your metric. If the last 7 days average is similar to the Optimizely average and above the long-run and similar week average then you have yourself a successful change.

Hope that all makes sense.. Looks like two good experiments there. Feel free to drop me a message if you want me to take a look at your site and recommend some possible tests for you.
David Shaw
Level 11
dfoley 06-12-14
 

Re: test shows revenue increase. how do I validate it?

It seems to be that I should just look at coversion rate for those periods you mention. As that takes into account traffic vs revenue right?

 

Yes, if you want to take a look and give me some suggestions i'd be glad to take them. www.sockdreams.com is our site.

 

We are working on a new home page with product listing, and we are working on a whole new mobile optimized site.

 

But it would be nice to test various aspects of the system NOW and then incorperate all that into the mobile site.

Level 2
dfoley 06-12-14
 

Re: test shows revenue increase. how do I validate it?

Also while i know that it's best that I stop testing for a week while i validate.. but i wish there was a way to continue testing. I want results sooner rather then later!
Level 2
shirley 06-16-14
 

Re: test shows revenue increase. how do I validate it?

adzeds,

Wondering how you set the benchmark you are referring to in your Thursday post...
Level 2
dfoley 06-16-14
 

Re: test shows revenue increase. how do I validate it?

The problem I am having is the the business is seasonal. When it gets cold people buy more socks. 

 

The conversion rate yearly curve hit it's high point in Dec, and it's low point in July.

 

This effects every sampling of conversion rates I do.

 

I can get all my samples from last year, but we are up from last year due to other factors.

 

If i take a long run average from 2 months before I ran the expirement, the conversion rate is higher.. because it was 2 months ago and that's how the conversion rate curve goes.

 

I'm still not sure how I can really get a good answer. Maybe the fact that conversion rate is not dropping but in fact 10% higher then the previous week is a good indicator? 

 

BUT.. i really really want some math. A formula i can plug this all into. Can you think of a way I could normalize by data by using last years conversion rate curve?

Level 2