How do I interpret the visitors remaining number?
The visitors remaining number we display in our dashboard is defined as the number of visitors you will need to reach a significant result, on average, if your baseline and variation conversion rates stay where they are. You can use this number to help decide when to stop your experiment, and when to keep it running.
This calculation takes into account all of the evidence you’ve seen so far in your experiment as well as the number of goals and variations you’re testing to show you the number of additional visitors needed to reach the significance level you’ve set in your project settings. Because the number is an average, the actual number of visitors you need could be a little higher or a little lower.
It’s also important to understand that the current observed conversion rates don't perfectly reflect the true underlying conversion rates of your baseline and variation. Observed conversion rates change by the day, hour, and minute due to random fluctuation. Yet while they’re not exact, they usually give a good ballpark estimate of what’s happening.
When deciding how long to run your test, keep in mind that the visitors remaining number is essentially a middle ground for how many more visitors you’ll need to see significance. If your conversion rates stay roughly as they are now, then you’ll need roughly that many visitors to reach significance. On the other hand, if the magnitude of your current improvement decreases, then you’ll need more visitors than is displayed. And if the magnitude of your improvement increases, you’ll need less.
You can also use the Optimizely sample size calculator to model a few different scenarios about how long you’ll need to wait with your current conversion rates and potential effect sizes. Then compare the sample size calculator’s output to the visitors remaining plus the number of visitors you’ve seen so far in your test.
Keep in mind though, the two visitor estimates won’t match up exactly. The in-product visitors remaining calculation takes into account how much evidence you’ve seen so far in your experiment, and how many other goals and variations you are currently testing.
Finally, we recently recorded a Webinar that runs through a few concrete scenarios of using the visitors remaining estimate to decide to stop or continue running a test. Here’s a direct link. The scenarios start at around 17:15.
Statistician at Optimizely