Rich-text Reply

Benefits of 1 optimization team vs. de-centralized optimization?

JohnH 01-17-16

Benefits of 1 optimization team vs. de-centralized optimization?

I'd love to learn how you've seen the most success at your organization. Do you have 1 team who is dedicated to optimization and helps all other product teams? Or, do you enable each product team to run their own tests? I'm thinking that it would be more effective with a dedicated optimization manager since that person would feel ownership for success, but some of my colleagues feel strongly otherwise. Thanks. 

Level 2

brianlang 01-18-16

Re: Benefits of 1 optimization team vs. de-centralized optimization?

tl;dr: I believe organizations will be most successful under a hybrid approach.


There are a number of benefits to having a dedicated optimization manager / team.


Knowledge of tools. Experience leveraging data and analysis for prioritizing tests. Understanding of best practices. A proficiency in statistics.


The last point is critical as it relates to a testing program - while it’s easy to produce numbers (the tools do it for you!), it’s very difficult to get numbers you can trust and be confident using when making business decisions. One needs to have a firm grasp of statistics to get to that level of confidence, and most individuals aren’t up to par (for example, how many individuals do you think understand the implications of repeated significance testing or making multiple comparisons [segments / goals / variations], understand the meaning of p-values, statistical significance / power, or have heard of Bayesian or Frequentist inference?).


Another benefit is those with a dedicated focus (in this case, optimization / testing), are in a unique position to be the internal advocate for said focus, empowered to promote learning and development across the organization – this is tremendously valuable! Just think, how amazing would it be if everyone utilized data [even to a small degree!] to influence their decisions and drive change Smiley Happy


One final (I swear!) benefit is upholding a standard with regards to test execution. This is a big one for larger organizations, where a business is typically split into multiple units (by product [such as a type of loan], or category [such as electronics vs. clothing]), who rely on a single property (think a website, such as to promote their line of business. While each unit has dedicated space (ex: a category or product page) there are shared spaces (such as the homepage) which typically attract many more visitors, and thus are considered prized real estate. If all business units are entrusted to execute their own tests without regard to how all other business units are executing their own tests, chaos will surely ensure (overlapping, conflicting experiments!)


There are a multitude of disadvantages to having a dedicated optimization manager / team.


Typically they focus on reporting, analysis, testing and optimization – there is a disconnect from the business as a whole (long term objectives and strategy, and individual tactics [such as product development, marketing campaigns, etc]). That’s not to say there is a total lack of awareness, but generally it is a much more limited view compared to others in the organization with a larger focus. Being even slightly disconnected will negatively impact ones’ ability to formulate meaningful hypothesis for new tests.


The above (focusing just on optimization / not having a direct connection to the product / P&L / business owner) causes issues with trust across teams, which limits partnership opportunities.


If you don’t know / aren’t close to my business, why would I entrust this super important task [growing/optimizing my business] to you?!” and “How can I be certain you won’t mess up my business in the process?!


Even if there is a leadership directive effectively stating, “this team/person is the optimization expert – leverage them as much as possible!” you will often be met with pushback from P&L owners - they will question your business acumen. This has potential to be incredibly detrimental to your organizational directive (optimizing!), as in most organizations whoever owns the P&L owns the decision making regarding changes to the website, mobile app, marketing channel and marketing collateral, etc. If you don’t roll-up to the P&L owner, you will have to convince them that your proposed changes (tests), and methodology for measurement, is more optimal than whatever they currently favor (be it their own experience and thoughts, their teams, other internal individuals / teams, agencies, etc). More time is spent trying to convince others than is spent on actually optimizing Smiley Sad. While there are certainly ways to address this, it’s an uphill battle, one that would likely be much easier if you were part of the team that rolled up to the P&L owner (ie, de-centralized).


A last point would be that centralized groups have a tendency to become silos, controlling everything coming in and going out. In many organizations, this can lead the team to appearing slow with engagements when there is more work to be done than there are resources available to support said work. If you aren’t proactively supporting a P&L owner (in a consistent, ongoing manner) in achieving their goals (through optimization), and your organizational structure doesn’t allow that P&L owner to execute tests on their own, you are prohibiting that P&L owner from exhausting all options for growing/optimizing their business. A P&L owner will rarely be an advocate on your behalf under these circumstances.


What is the optimal route to take?


My advice is to not structure this question as, “What’s better, one central team, or a de-centralized team(s)?” and instead consider a third option… a hybrid between the two! An organization’s most valuable asset is it’s people – figure out who is good at what and empower them to add value to a optimization program! Sure, it makes sense to centralize certain components (things like interpreting results, standardizing the constraints of how an experiment can be setup an executed, etc), and de-centralize other components (such as forming hypothesis, leveraging input from individual teams on what a meaningful change to their business would be, etc).


Hope this helps!

Brian Lang
MartijnSch 01-21-16

Re: Benefits of 1 optimization team vs. de-centralized optimization?

In our case we've decided that we could work best with an optimization team. As a tech publishing company we don't really have products, we just have a big news site where we post the content on. Because of this we don't really have product teams but work with a central team on all the projects we have. The team is set up in such a way that we have specialists and they're supporting the project teams whenever they can next to the central teams goals.

At some point you also want to mature the whole organisation or the knowledge in general so that's why having a central team can be more efficient as well as it will have the ownership for optimization and is able to push deeper knowledge across an organisation.
robertchan 01-21-16

Re: Benefits of 1 optimization team vs. de-centralized optimization?

As a media company, we work very closely with relevant groups to determine what to prioritize and optimize for conversion. This includes working closely with product managers, the web development team, edit, and graphics. I find that combining the best of all departments creates the best environment for ideation and follow up post-execution.
Robert Chan

Experimentation Hero

Re: Benefits of 1 optimization team vs. de-centralized optimization?

The answer is different based on the size of your company and the volume of optimization you want to do. If you think about your optimization program through a multi-year lens, I'd suggest your best end state would be a world where optimization is decentralized. In this end state many people throughout the organization feel comfortable experimenting and reading results. At this point your organization is learning across many different teams at high volume.

However, in order to reach this state described above, you'll probably need a centralized team. This centralized team can go deep on building the processes and systems to allow many people to experiment. My advice would be centralize the team with a long term goal of decentralization.