Ask the Expert: Hudson Arnold, Sr. Strategy Consultant, on Adopting an Experimentation Philosophy
The live Q&A portion of this AMA is closed. However, I would encourage you to read through all the amazing content that has been discussed here.
Respond to this post with questions you may have, bounce ideas around, and get to know the expert!
Hi Optiverse! I’m Hudson Arnold, Sr. Strategy Consultant at Optimizely. I’ve learned a lot about optimization and the importance of experimentation over the years. Ask me anything!
Most recently, I've written a paper talking about what exactly makes the world's best optimization organizations so successful - 'Adopting an Experimentation Philosophy: Closing the Gap to a World-Class Optimization Practice' - which you can download here. This paper will help you remove bottlenecks in your optimization program with a clear optimization methodolgy, boost creativity in your ideation process, and set your team up for success by accurately benchmarking your program.
For this Ask the Expert, feel free to pose your questions concerning:
- Formalizing an optimization methodology for your organization
- Defining goals for your optimization program and aligning them with your business
- Creating business intelligence reports to feed your ideation process
- Benchmarking your operational metrics
- Any question you have about your experimentation strategy!
You can ask questions until noon PST on Thurdsay, March 1 at which point this session will be closed. Check out the post here for guidelines and additional information.
If you are interested in being featured as an Expert for a specific topic, please email email@example.com
I work with many SaaS teams that want to mature their optimization programs and start experimenting within their company's product, but the product teams are often times hesitant to do testing with a client-side solution like Optimizely. What's the best way to get buy-in from product or more technical teams so that they can start testing across their entire website?
Hey @jason-gsell long time no talk.
Hey Jason, long time no talk.
For product teams that want to experiment in the most fundamental parts of their application, not just front-end variables like layout, look and feel, messaging, etc, the new Optimizely Full Stack solution should be considered.
For the technically minded, I recommend starting your quest at developers.optimizely.com! For those looking to understand more about what Full Stack is and how it can be used, I'd look at one of the recent webinars we've published.
For some background on what Full Stack is and why we've built it:
Optimizely has historically thrived on democratizing experimentation processes that the leading technology companies produce with custom engineering time on their front-end experiences (like Amazon and Google), and we couldn't be more excited to continue that journey deep into the workings of core application functionality (like search algorithms, pricing, and experiences within cool cross-channel experiences like Alexa, chat bots, in-store kiosks and more).
There's also some really cool blog posts by the likes of Facebook (links to their in house AirLock native mobile testing framework and PlanOut experimentaition library), Pinterest (link to their explanation of what they built for their experimentation framework), and Netflix (Their in house solution), that discuss in detail challenges and opportunities that experimentation within their application can present. These companies have invested inordinate amount of money and engineering time on building and maintaining these systems (often times building custom code libraries, event ingestion and reporting dashboards, and employing teams of analysts skilled in SQL, R, etc to report on), which is a good indicator of the value that we hope to democratize by producing and supporting Optimizely Full Stack. We've been thrilled with the initial reception and can't wait to see the further progress that product teams will make by more easily and efficiently running experiments on our Full Stack product.
Maybe you can tell how excited I am
Hopefully this helps you get started in your journey!
Experimentation culture is certainly evolving. That evolution, and helping evangelize and improve it, is probably the top reason why I'm so excited to come work at Optimizely every day.
There are some trends that are accelerating right now that I'm deeply optimistic about because I'm seeing evidence of them every day, in our data and in my experiences working with Optimizely customers and partners!
Here are the changes and trends in experimentation culture I'm most excited about:
- Experimentation Grows Up! : In the past (and to some extent still today), experimentation would often be an add-on, in which some landing pages would be tested through, or a media team would test headlines and different direct mailers (David Ogilvy was doing this 50 years ago). What we're increasingly seeing is that the mantras 'always be testing', 'everything is an experiment', along with the technology trends that are making it easier, is that companies are completely disrupting themselves by truly making experimentation part of their DNA. Plenty of Optimizely case studies illustrate this (I like the webinar I did on 'building a center of excellence' with Atlassian that we put out a few months ago. Check out that session and others on the Virtual Summit on demand page - there's really strong content here. )
The exciting part about this is that some of the long-held promise of experimentation is certainly coming to bear. More dramatic and interesting kinds of tests are being launched all the time, and the professionals that have built careers in the space are quickly growing and adding value at ever stronger companies. The investment we're seeing from companies like IBM, who are disrupting themselves internally to transform their marketing into an experimentation-driven culture, and smaller companies like Bonobos, who are leading their space in many ways by driving an optimal customer experience, is incredibly inspiring.
- Cross Channel: We just spoke a bit about Full Stack, which has immediate promise for every product and digital marketing org in its ability to run server side experiments.
Yet, the untapped potential here is something I've been dreaming about for years: experimenting in the world of connected devices, or bringing experience optimization into the real world.
What does this mean? Here are some immediately possible kinds of experiments that companies can run by instrumenting a Full Stack experiment:
In-Store Digital Signage: Companies like McDonalds (in their drive through windows), Airlines (in their Airport kiosks) and fashion retailers, use digital signage in physical transactional locations to drive revenue. WIth Full Stack, its entirely possible and advisable to experiment on the conversion aspects of these experiences!
Here are some cool digital signage that could be made to convert better through experimentation; what ideas for improvement could you imagine running as an experiment?
McDonald's ordering Kiosk
American Airlines Kiosk
In Store Promotion at a Fashion Retailer
Digital menu at a juice shop
Now, these are only some superficial examples of running experiments on a digital display that's pretty similar to a laptop or phone.
If you consider some of the variables that control key applications BEHIND the scenes that Full Stack can help experiment on, you'll see the potential is truly, massively inspiring:
- Traffic Light timing
- Car pool logic algorithms on Uber
- Discounting logic for CRM campaigns
- Bundling strategy for B2B and B2C companies alike
- Agricultural nutrition and yield optimization
- Chatbot experiences
...and more...tell me Robert, what are you excited to optimize?
- Integration Sensation: Still, Optimizely
has a long way to go to properly integrate with the CMS, internal service, and datasets that companies use.
Particularly when it comes to Personalization, the potential that is offered by not just connecting into companies' proprietary data (Which we can do via integrations or Dynamic Customer Profiles), but into intelligence services like IBM Watson, which are beginning to transform operations of businesses all around the world.
Recently our CEO gave a demo for IBM's marketing group that illustrated how Watson could help assess whether a given experiment design was likely to produce a winner! Amazing stuff and it shows the power we're beginning to tap into by helping connect companies' data sets into tangible, actionable strategy.
Robert, I hope this is interesting and inspiring to you as you continue to rock it - and thank you for being such an important part of our Optiverse community.
It seems like the stats engine is sophisticated, but leaves potentially unintended questions/results that might occur impossible to explain.
I'd be interested if you encountered problems with colleagues IT/Engineering in the past and if so how did you overcome those obstacles?
Any general strategy that you could recommend so that product management and IT are better aligned in terms of experimentation technology and how deep it should go within a given framework?
This is an interesting topic!
First, I want to acknowledge that today, many of the kinds of analysis you may want to do in an exploratory mentality, like ad-hoc segmentation on all of your sites historical data, are definitely best done outside of Optimizely. These kind of reports can be helpful for understanding your users and inspiring experiment hypotheses.
But, when it comes to making decisions on your experimental data, Stats Engine has some significant advantages. Optimizely lets you assign Primary, Secondary, and tertiary metrics, which helps you reach statistical significance faster, and uses a unique hybrid of Frequentist & Bayesian approaches that is optimal for helping you achieve confidence in your statistical decision making.
An experiment is so valuable, because for the audiences & experiences being tested, you'll have no more rigorous way of making actual decisions about how to improve performance. Well designed experiments in Optimizely can generate tremendous insight, often times more than reviewing historical data in other tools; when you can compare how different segments react to the same message in real time, you can generate much richer whys (strategic observations about the kinds of experiences and messages that valuable to iterate on).
Optimizely has effective approaches for integrating experiment variation data into your dedicated analytics solutions, which is a great way to blend the best of both worlds.You also have the ability to reference Stats Engine declarations by accessing Results in the REST API. Many companies find value by both using Stats Engine for Experiment decision making and further exploratory analysis by integrating into their analytics solution, and this blended approach is what I would recommend most companies do.
But, the best approach varies by team, digital maturity, and resources. What has been working for you?
Super valid line of questioning here. Who hasn't heard tales from IT, Marketing, Product orgs butted against each other:
"IT is making it impossible to innovate."
"Product isn't following a proper deployment process and that is a risk"
And there's some truth to this, particularly for bigger organizations. And all of these groups have very healthy, smart reasons for wanting to push things in different directions. A focus on quality and security that IT brings is super important. But moving quickly and breaking things is often the best way for product teams to work.
Yet, what's super exciting is that some of the other trends in this thread (full stack, personalization, maturity of experimentation culture) are very real, and we're seeing evidence with some of our larger partners right now that Optimizely is proving to be a rare initiative that brings these different groups together.
1. Executive Alignment
The simplest thing to get something done in this front is to get executive alignment. May seem like a cop-out answer, but its really true. The most mature experimentation organizations, the companies making the most rapid and confident moves in the space, have mandates from the senior executive level to transform their operations into launching EVERYTHING as an experiment. What we've had success with recently is transforming this mandate into an operational plan to tackles some of the most important pre-requisites for cross-functional experiments:
- Standardizing on experimentation tech; this is a big decision and often needs to come at the CTO, CMO, COO level
- Training; all these groups will need to understand the intricacies of the technology and how it will fit in with the existing stack
- Methodology; One of the most fundamental aspects to rolling out experiment programs between different functions is to map out the process of experiment ideation, design, roadmapping, development and analysis. Who owns which step of the methodology? How does this process change for back-end experiments vs. front-end? For product vs. marketing team?
So these are the outcomes of executive alignment. But getting there can be hard. The MOST important thing is to speak to these strategic leaders is a language they understand, in the universal language of business: Data.
- What has the impact of experimentation been at your company? At other notable brands?
- Can you extrapolate the value of additional experimentation into new workflows that may currently be managed by IT / Eng? In the term of more efficient production, or risk mitigation?
- What would an additional # of experiments (that you could argue are being constrained by interactions with Eng/IT) do to the bottom line for your org this year? This is hard logic to ignore! Might make it easier to get alignment.
2. Make it Easy, and Extend the Olive Branch:
Now, there is also plenty of great work to be done by working directly with the teams that are currently tough to work with. In my mind, it comes down to making it easy, obvious, and attractive for those teams to work with yours!
Clarify the Problem:
These engineering and IT people are super smart. They have built a career on distilling complex problems into pure logic. Speak to them in their language.
Map out the development process of product and eng teams in separate workflows. Pinpoint where experimentation plays a part; roadmapping, prioritization, and additional steps to development. Work together in a meeting or workshop to rigorously map out where potential issues could arise; state your goal and let these people help you; if we can all agree that doing more experiments is a good thing, what are some things we could do to better align and adopt it in these processes? Get the challenges properly laid out on a white board so that there is 0 ambiguity about the challenges being faced, and see how your teams respond.
In my experience, people will love to solve problems for you if you can properly define it and convince them its a problem worth solving.
Evangelizing experimentation as a practice that they can use too:
- What are some experimental concepts that Eng & IT could use in their businesses to advance their goals? We've ran experiments here at Optimizely that empirically quantify the page load differential b/w different versions of our results page, and we're working with companies that are experimenting with tough architecture decisions. Can you work with that team to identify the most difficult decisions they have to make? Can you develop an exiciting experimental concept to trial with Full Stack?
Remove Fear, Uncertainty, and Doubt:
- What are the specific concerns of your IT/Eng neighbors? Is it scarcity of resources? Is it performance or reliability concerns? QA?
In my experience, these concerns can spring from a place of fear and not having all the context; these people are smart pros, and Martech is infringing on IT's traditional home base all the time.
Be a good neighbor to these people! More precisely determine what their issues are, and proactively address them.
- What are the best in class QA practices your team is going to use to ensure quality control is never an issue?
- What is an approval process you could propose that efficiently gets experimental concepts approved once a QUARTER, instead of a new approval for every experiment?
- How can you evangelize experimentation as an optimal use of making data driven decision making, that can help inform that team's roadmap and avoid costly architecture decisions, ultimately making their jobs more interesting and safer?
- How can your team work with IT and experts from Optimizely to improve your technical implementation to minimize page load performance? Can you start measuring performance and reliablity as a tertiary metric, and work with IT to make it better quarter over quarter?
Bring Teams Together
Simple alignment exercises can be amazingly helpful.
- Buy pizza and invite members from IT to learn about what your team does, how it works, and what it accomplishes
- Ask their team lead for a representative to be an internal SME, a liason between their two teams
- Invite their team to contribute ideas to the testing roadmap
So, a lot here, and how to move forward certainly depends on your specific situation. @maddy, does any of this ring a bell for you? What would you do differently going forward? What doesn't seem to work for you?
Let us know. This kind of conversation makes us all smarter.
thank you very much for your extensive answer.
All the topics you mentioned ring a bell for me. From my experience the backing of a top C level sponsor is the most important thing. This can vary depending on the type of company and the way it is structured (flat vs hierarchic).
The other parts seem to easy so that they are often neglected form my experience. But they are also very important of course.
And all actions / efforts have to be continnous.
To scale things up (talk about the # of experiments per months/week..) I made good experience teaching product managers the methodoly and giving hints on how to "justify" the time they then spend on experimenting (if that is needed, but often that's a concern I hear from them). Once they made a "win" it becomes like a habit for them and they also have now great data to show to the management and their team which boosts their career and therefore more and more colleagues in the organisation come onboard.