Rich-text Reply
Highlighted

AB Testing Brown Bag to showcase work. Has any one else done this?

mstreegan 10-06-14

AB Testing Brown Bag to showcase work. Has any one else done this?

I'd like to host a company-wide AB Testing Brown Bag to educate and showcase the AB Testing work that my team has done. We're excited to start spreading the word on results and why we're making the decisions that we're making. Plus I'm hoping that this will be a catalyst to gain more support whilst spreading our AB testing culture.  

 

I'm thinking 45 mins, 30 mins of results sharing and QA plus 15 mins for future tests and wrap up.

 

Has anyone tried this? How'd it go? Format that worked/didn't work?

 

Thanks!

 

Amanda 10-07-14
 

Re: AB Testing Brown Bag to showcase work. Has any one else done this?

Excited to see how the community responds here. I know @MJBeisch has done a similar lunch meeting where they brainstorm ideas for upcoming tests. I also know @MeganBush has led some very successful trainings on Optimizely. These are both slightly different scenarios than the one you're proposing, but I imagine a lot of the best practices will carry over! Can you guys provide some tips on what's worked/hasn't worked for you?

Optimizely
adzeds 10-07-14
 

Re: AB Testing Brown Bag to showcase work. Has any one else done this?

I really love the concept of this.

We used to hold a meeting every Friday where we would discuss the testing that had been done, share results, get feedback and discuss possible extension or new variations to test.

This included people from up and down the hierarchy and allows people visibility into what is going on and helps to share ideas to other departments who can learn from our test findings.
David Shaw
Level 11
Amanda 10-07-14
 

Re: AB Testing Brown Bag to showcase work. Has any one else done this?

Thanks for sharing @adzeds - can you share the specific format you used? Was there anything that went over especially well with the group?

Optimizely
MeganBush 10-07-14
 

Re: AB Testing Brown Bag to showcase work. Has any one else done this?

I've done a couple of different trainings and testing "showcases" for our company internally.

I definitely think starting with results sharing is the way to go! Nothing gets the team more excited than hearing about actual successes and learnings from test results (especially when the results include added revenue for their company or clients!). In my experience, after the team asked quite a few questions about the results, their ideas didn't necessarily start flowing immediately. The ideas slowly trickled in through emails and people approaching me the following days, weeks, etc. as they started to think more about testing. Perhaps sending a follow up email (survey/google form/etc.) a couple of days after your presentation could generate more ideas from the team.

And definitely don't let them forget! Keep presenting test results, both good and bad, to remind people that they should always have it in the back of their minds when they are suggesting/making changes to a website.

As Amanda mentioned, I've also hosted training sessions for our developers and account managers. The biggest win in this training session was not showing them how easy it is to use Optimizely, but showing them the incredible capabilities the platform has. Reviewing the variety of features the platform has, as well as the customizations it is capable of and providing examples of how our clients could use the more complex ones definitely intrigued every one on the team. Similarly to the previous showcase I mentioned, ideas started flowing the more people thought about these features in the coming days and weeks after the training.

Best of luck in sharing with your team! Let us know how it goes and what works for you!
Megan Bush
Hudson 12-02-16
 

Re: AB Testing Brown Bag to showcase work. Has any one else done this?

[ Edited ]

Hey @mstreegan!

 

I've done many of these kind of meetings before. While the best content and format depends on what your goals are, I'd suggest something along the following lines. 

 

1. The Hook / Which Test Won? (5-10 minutes)

 

This classic communication tactic is an invaluable introduction into A/B testing. By presenting known case studies like the ones Optimizely publishes, and asking the audience to guess, which variation one, you'll activate intellects and inspire curiosity. 

 

For this section try showing

  • 2-3 case studies (they can be public facing ones; known company logos are an effective means to build interest and provide social validation)
  • For each case study, introduce with a bit of background and show two variations that were tested in an experiment
  • Ask the crowd by a show of hands to cast their votes as to which variation performed better
  • Followup with a couple people to ask 'why'
  • After showing the results, illustrate the 'best known' hypothesis of why that variation one and re-iterate that everyone's guess was a good hypothesis

 

2. What is A/B testing? Why is it valuable? (5-10 minutes)

 

For those less familiar with the space, or even a group with mixed levels of experience, its important to provide context into the industry trends. Think of how you can answer the following questions a beginner might have

  • What is A/B testing? What's its history?
  • Why is it valuable?
  • What role does Optimizely technology vs. people like you and your team play in doing it? What's the process?
  • Consider using resources like the Forrester Continous Optimization report, and testimonials from industry leaders, to back up the story with solid data points.

3.  A/B testing at your company (20 minutes)

 

A writing teach of mine used to evangelize PUP, or 'personalize -> universalize -> personalize. After branching into the industry context, bring it back to the story you really want to tell: "results and why we're making the decisions that we're making. Plus ... a catalyst to gain more support whilst spreading our AB testing culture."

 

In this section, show

  • The team doing testing and their roles
  • The goals and charter of testing; how do they fit into the company's larger goals? (Consider using a Goal Tree)
  • Timeline of testing at your company; when did it start? What changes have come? How has the process evolved.
  • Cool results! The case studies, metric improvements, risk mitigated, and insights generated by testing thus far
  • What's next; goals for the program, getting more people involved, 2017 goals!

4. Q&A + Next Steps / A Call to Action

 

Whether its contributing test ideas to a centralized submission form, incorporating experimentation to projects or teams that currently aren't, or even suggested reading, let people know how they can contribute and invite them to!

 

----

 

Now this isn't a paragon for such sessions, so please exercise your creativity and passion! Does this idea make sense to you? What else would you consider adding?

 

Happy Testing,

Hudson

 

 

Optimizely

Re: AB Testing Brown Bag to showcase work. Has any one else done this?

excellent idea, I've been trying to make something like that
but in spanish
SEM and CRO Specialist