Benefits of 1 optimization team vs. de-centralized optimization?
I'd love to learn how you've seen the most success at your organization. Do you have 1 team who is dedicated to optimization and helps all other product teams? Or, do you enable each product team to run their own tests? I'm thinking that it would be more effective with a dedicated optimization manager since that person would feel ownership for success, but some of my colleagues feel strongly otherwise. Thanks.
tl;dr: I believe organizations will be most successful under a hybrid approach.
There are a number of benefits to having a dedicated optimization manager / team.
Knowledge of tools. Experience leveraging data and analysis for prioritizing tests. Understanding of best practices. A proficiency in statistics.
The last point is critical as it relates to a testing program - while it’s easy to produce numbers (the tools do it for you!), it’s very difficult to get numbers you can trust and be confident using when making business decisions. One needs to have a firm grasp of statistics to get to that level of confidence, and most individuals aren’t up to par (for example, how many individuals do you think understand the implications of repeated significance testing or making multiple comparisons [segments / goals / variations], understand the meaning of p-values, statistical significance / power, or have heard of Bayesian or Frequentist inference?).
Another benefit is those with a dedicated focus (in this case, optimization / testing), are in a unique position to be the internal advocate for said focus, empowered to promote learning and development across the organization – this is tremendously valuable! Just think, how amazing would it be if everyone utilized data [even to a small degree!] to influence their decisions and drive change
One final (I swear!) benefit is upholding a standard with regards to test execution. This is a big one for larger organizations, where a business is typically split into multiple units (by product [such as a type of loan], or category [such as electronics vs. clothing]), who rely on a single property (think a website, such as amazon.com) to promote their line of business. While each unit has dedicated space (ex: a category or product page) there are shared spaces (such as the homepage) which typically attract many more visitors, and thus are considered prized real estate. If all business units are entrusted to execute their own tests without regard to how all other business units are executing their own tests, chaos will surely ensure (overlapping, conflicting experiments!)
There are a multitude of disadvantages to having a dedicated optimization manager / team.
Typically they focus on reporting, analysis, testing and optimization – there is a disconnect from the business as a whole (long term objectives and strategy, and individual tactics [such as product development, marketing campaigns, etc]). That’s not to say there is a total lack of awareness, but generally it is a much more limited view compared to others in the organization with a larger focus. Being even slightly disconnected will negatively impact ones’ ability to formulate meaningful hypothesis for new tests.
The above (focusing just on optimization / not having a direct connection to the product / P&L / business owner) causes issues with trust across teams, which limits partnership opportunities.
“If you don’t know / aren’t close to my business, why would I entrust this super important task [growing/optimizing my business] to you?!” and “How can I be certain you won’t mess up my business in the process?!”
Even if there is a leadership directive effectively stating, “this team/person is the optimization expert – leverage them as much as possible!” you will often be met with pushback from P&L owners - they will question your business acumen. This has potential to be incredibly detrimental to your organizational directive (optimizing!), as in most organizations whoever owns the P&L owns the decision making regarding changes to the website, mobile app, marketing channel and marketing collateral, etc. If you don’t roll-up to the P&L owner, you will have to convince them that your proposed changes (tests), and methodology for measurement, is more optimal than whatever they currently favor (be it their own experience and thoughts, their teams, other internal individuals / teams, agencies, etc). More time is spent trying to convince others than is spent on actually optimizing . While there are certainly ways to address this, it’s an uphill battle, one that would likely be much easier if you were part of the team that rolled up to the P&L owner (ie, de-centralized).
A last point would be that centralized groups have a tendency to become silos, controlling everything coming in and going out. In many organizations, this can lead the team to appearing slow with engagements when there is more work to be done than there are resources available to support said work. If you aren’t proactively supporting a P&L owner (in a consistent, ongoing manner) in achieving their goals (through optimization), and your organizational structure doesn’t allow that P&L owner to execute tests on their own, you are prohibiting that P&L owner from exhausting all options for growing/optimizing their business. A P&L owner will rarely be an advocate on your behalf under these circumstances.
What is the optimal route to take?
My advice is to not structure this question as, “What’s better, one central team, or a de-centralized team(s)?” and instead consider a third option… a hybrid between the two! An organization’s most valuable asset is it’s people – figure out who is good at what and empower them to add value to a optimization program! Sure, it makes sense to centralize certain components (things like interpreting results, standardizing the constraints of how an experiment can be setup an executed, etc), and de-centralize other components (such as forming hypothesis, leveraging input from individual teams on what a meaningful change to their business would be, etc).
Hope this helps!
At some point you also want to mature the whole organisation or the knowledge in general so that's why having a central team can be more efficient as well as it will have the ownership for optimization and is able to push deeper knowledge across an organisation.
However, in order to reach this state described above, you'll probably need a centralized team. This centralized team can go deep on building the processes and systems to allow many people to experiment. My advice would be centralize the team with a long term goal of decentralization.