Rich-text Reply

Creating a strong hypothesis for your experiment

Allison 11-17-14

Creating a strong hypothesis for your experiment

[ Edited ]

When A/B Testing it is super important to create a strong hypothesis to get the most out of your experiment! One common mishap in setting up your experiment is not making your hypothesis specific enough to learn from it *regardless* of the outcome. You also want to be sure you can take those learnings and apply them to your organization.


E.g. we tested two subject lines against each other for an email:

- FAQ: How long should my test run?

- Are you measuring your tests accurately? Measuring statistical significance of your test


While we were excited to have discovered that we reached a statistically significant winner (FAQ won), we are forced to iterate on this test, because we really can't specify exactly what it was that made it be the winner. Does our audience like FAQs? Did they like that it was in first person? Was it because of the all caps?


I love this post that Shana Rusonis wrote about creating a strong hypothesis, so I thought I would share here to avoid these common pitfalls!!


Key items to look for in creating a hypothesis: 


A learning opportunity, regardless of outcome

Hypothesize for every outcome

Build data into your rationale

Map your experiment outcomes to a high-level goal

Document your hypotheses.