Creating a strong hypothesis for your experiment
When A/B Testing it is super important to create a strong hypothesis to get the most out of your experiment! One common mishap in setting up your experiment is not making your hypothesis specific enough to learn from it *regardless* of the outcome. You also want to be sure you can take those learnings and apply them to your organization.
E.g. we tested two subject lines against each other for an email:
- FAQ: How long should my test run?
- Are you measuring your tests accurately? Measuring statistical significance of your test
While we were excited to have discovered that we reached a statistically significant winner (FAQ won), we are forced to iterate on this test, because we really can't specify exactly what it was that made it be the winner. Does our audience like FAQs? Did they like that it was in first person? Was it because of the all caps?
I love this post that Shana Rusonis wrote about creating a strong hypothesis, so I thought I would share here to avoid these common pitfalls!!
Key items to look for in creating a hypothesis:
A learning opportunity, regardless of outcome.
Hypothesize for every outcome.
Build data into your rationale.
Map your experiment outcomes to a high-level goal.
Document your hypotheses.