Rich-text Reply

Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

JessieBecker 02-08-15

Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

[ Edited ]

Jessie Becker.jpgThe live Q&A portion of this AMA is closed. However, I would encourage you to read through all the amazing content that has been discussed here. 

 

Hi Optiverse! I’m Jessie Becker, the Chief Marketing Officer at Optimizely. I’ve learned a lot about marketing and the importance of testing over the years. Ask me anything.

 

At Optimizely, I oversee all marketing activities including demand generation, customer marketing, product marketing, content marketing, and PR.  Previously, I was the Interim Chief Marketing Officer at Netflix, where I was responsible for the global brand and customer acquisition programs. During my twelve years at Netflix I held a variety of marketing positions with increasing responsibility, and led a cross-functional product management, consumer insights, and engineering team to optimize all Netflix signup flows (domestic and international; web, mobile and 10’ UI), with annual gross signups exceeding 10 million.

 

I’ve worked at Amazon.com, Microsoft, and Oracle. I have a B.A. from the University of Pennsylvania, a B.S. in Economics from The Wharton School, and an M.B.A. from Stanford University, where I was an Arjay Miller Scholar.

 

I live in Palo Alto, California, with my husband and two children.

 

----

 

Tweet this "Ask the Expert" discussion.

 

You can ask questions until EOD February 16, at which point this session will be closed.  

 

If you are interested in being featured as an Expert for a specific topic, please email optiverse@optimizely.com

--
Chief Marketing Officer at Optimizely
Optimizely

greg 02-09-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hey Jessie, can you tell us about the thought process and reasoning for switching your value prop from A/B testing to experience optimization? Was that just a change in messaging, or in overall strategy? What led Optimizely to that decision?

Level 2

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Greg,

 

Thanks for noticing!  We're really excited about the change.  

 

Experience optimization is a reflection of how many of our customers are already using Optimizely day to day.  A/B testing -exploring which variation performs best, and then making a decision based on that exploration - is a key componen of any optimization programt, and something that we think is critical to getting great business results.  Experience optimization goes broader to include continual customization of what customers/visitors are presented, based on what you know about that person.  What you know could include prior visits to your site/app, traffic source, third party information that you've enhanced your records with, or a variety of other things. 

 

Many of our customers already use Optimizely this way to customize how their site or app renders for different audiences.  Many customers use A/B tests first to understand what variation works best for a specific audience, and then use targeting to deliver that optimized experience to 100% of a specific audience.   

 

Jessie

 

 

 

 

 

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hey Jessie!

 

Congrats on the new role!  I worked with Michael Zeman on the Netflix team to launch our relationship with MEC, and currently I lead sales for a fellow YC company called Virool.  We are a video distribution platform that has done really well with B2C, but now we're building out a team focused on B2B initiatives.  I'm curious how big of a role digital video plays for Optimizely, especially since I'd imagine most of your digital spend is on DR / UA initiatives, while video tends to be more top funnel?

 

Thanks (in advance) for taking the time to reply and good luck! Smiley Happy

 

 

Phil 02-09-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

So, I am running a test and each segment is reacting differently. Search segment 50% gain but moble had a 6% loss. Would you recommend creating segmented tests or go with the overal. The overall results show an 8.9% increase.

Level 1
angelscup 02-09-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hey Jessie! Perhaps outside of the scope, but what changes to my site do you think I should test? angelscup.com

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

[ Edited ]

Hi Jessie!  Thanks for taking the time to chat with us. 

 

I would love to learn more about how you structured and scaled the cross-functional optimization team at Netflix.  

 

For example, how did the cross-functional team of product management, consumer insights, and engineering work together? Were there sub-teams working on specific metrics, user flows, products, regions, etc?  What type of roles were in these teams?  How did you support international initiatives, localization, web, mobile, etc?   What worked?  What didn't?

 

Thanks!

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi,

I checked in with our internal team and they let me know you posted Thursday, we responded immediately that we are investigating, and they expect to get back to you today.  Hopefully that answers your question?

 

For others who may be interested - like many other companies, we have established a bug bounty program to help us identify and quickly respond to any issues.

More detailed information is available here: https://www.crowdcurity.com/optimizely

 

thanks,
Jessie

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Jessie,

 

Cool to ask you questions.

Let me set the scene for you.

 

I evaluate our marketing effectiveness based on 15 data metrics and specific marketing scorecards to each campaign alongside with Business Revenue and retention.

 

So what I am interested in is how you evaluate the effectiveness of your marketing campaigns and how come you do it the way you do.


Regards

Josh

------
Josh always rolling with the changes

"You must be shapeless, formless, like water. When you pour water in a cup, it becomes the cup. When you pour water in a bottle, it becomes the bottle. When you pour water in a teapot, it becomes the teapot. Water can drip and it can crash. Become like water my friend.”

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Jessie,

 

Thank you for doing this. We finally have our site where we want it (amazing member feedback, phenomal results and an easy to use training program, decent analytic scores), but we are not consistantly getting conversions. 

 

We have even had surges of signups, but then long draughts. I would literally do a back flip if we could just get to a 2% conversion rate daily. Our traffic is increasing, and people seem interested, but our signups are not there.

 

My question is, do you have team members that can do the split testing for us? I'm too overwhelmed with other tasks to take this on. I know your program is supposed to be easy to use, but I'm at the point where I just want to hire the best.

 

Thank you for your time. 

 

Justin Rundle

Workout Anywhere

http://www.workoutanywhere.net 

allanminium 02-09-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Jessie,

Thanks for doing this AMA.

One of the biggest challenges I have is the new mixture of Marketing and Product divisions. I'm in marketing and the role of AB testing/experience optimisation falls under my job description, but product are constantly butting in to say I should not be altering their product roadmap. Have you encountered this situation and how did you resolve it?

Appreciate your time!


Allan

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Phil,

 

There are a number of paths you could take with these results. If you are noticing a difference in high value segments and you have the resources to throw at targeting a segment, then you'll probably want to take the learnings and turn them into a personalization campaign. A personalization campaign is just another experiment that isn't really an experiment, but targets each segment to deliver the winning variation. For example, create a new experiment, delete the variation so that you have only the control (which receives 100% of traffic), set the audience targeting to include the Search segment, but exclude the Mobile segment, and copy/paste the variation code from the other experiment. This will allow you to deliver the winning experience to the Search+non-Mobile segment, but not show the experience to the Mobile segment.
 
As you can see, maintaining that kind of personalization can make things a bit complicated for future experiments so ensure that you have the resources to support this kind of strategy before going down that path. The question is essentially, is the 50% gain in Search non-Mobile visitors worth the effort required to maintain the personalization? If yes, do the personalization. If not, then you may just want to deploy the winner to all traffic since it gave you an 8.9% increase.
 
thanks
Jessie
--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Sahil,

 

Nice to meet you!  Michael Zeman and I didn't overlap at Netflix unfortunately.

 

Regarding your question about digital video - I'd say it's an area we are excited about but isn't yet proven out.  We think video is a really compelling way to communicate ideas, and we use it a lot for customer stories that we can then share widely on our social networks and with our customers directly.  From a strict DR perspective we haven't yet seen it prove itself as an efficient way to spend our advertising dollars.

 

Hope that helps!

Jessie

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi angelscup,

 

Well, yes, it's a little outside the scope - but my quick feedback is that the highest impact thing you could test would be a free trial.  You can still collect credit card information upfront and have it auto-renew, but free trial will make a huge difference in your response rate.

 

Besides that, I'll point you to some of our other resources for testing ideas:

 

http://blog.optimizely.com/2014/02/27/21-ideas-to-brainstorm-your-next-ab-test/

 

Good luck!

Jessie

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Justin,

 

Congratulations on getting your site where you want it -- that is fabulous!

 

Re testing and increasing your conversions - if you've reached the point where you need a little extra help, I suggest you check out our awesome solutions partners. You can find them all at https://www.optimizely.com/partners/solutions/.  

 

Hope that helps,

Jessie

 

--
Chief Marketing Officer at Optimizely
Optimizely
angelscup 02-10-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

@JessieBecker - Interesting, thanks! Have been thinking about doing something like that. This might be what we need to push us over the hump.

pcuci 02-10-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

[ Edited ]

What is the minimum number of unique visitors required to hit a landing page for A/B tests to be meaningful? How to pick the threshold for different types of pages?

 

10/day, 1000/day?

 

Thanks!

 

Paul Cuciureanu

http://Enfluence.io - Digital High Blood Pressure Program
Join a team, get coached, and lead a healthier life!
Level 1
MartijnSch 02-10-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Jessie, I got a question more related to your role as CMO then in particular on A/B testing. How you do divide your time between your team, the outside world and improving the product/ company and how do you try to spend that time and how do you come to these decisions?
bd 02-10-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hello there.

A question that I am constantly battling is when we are running sale via promo code discount is do we plaster the banner on teh website so that everyone and anyone see's it and potentially could recive a discount if they convert. 

The argument is why eat into our net for no reason. If the customer was going to buy anyway why give away the free code?

 

The sale in general is blasted out to our email list of approx. 200-300K. We don't prospect with these promos at all.

 

My feeling is we set up the banners so they are displayed to new visits only. We run it so that 50% of new visitors get the banner and 50% of new visitors don't. Teh neasurment stradegy can be
A) New Customer Aquisition
B) Total Orders
C) RPC

 

Your thoughts? 

bd
Level 1

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Thank you for the reply. Is the star rating based on the level of jobs these partners will accept, or their experience level?
greg 02-10-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

[ Edited ]

Hey Justin, if agencies are a bit too impersonal and/or expensive for you, anoher option is to work one-on-one with a consultant who'll take the whole optimization process off your hands. Smiley Wink

 

Click the link in my signature if you're interested.

Level 2

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Ryan,

 

The way we set it up is that every product owner also had responsibility for testing and optimization of their area - they were integrated, not separate.  I focused on the non-member experience - the path for getting people signed up for Netflix.  The key metrics we tracked were signups per visit, total number of signups, and retention (of people who started with Netflix, how many renewed after their first month). 

 

Each product owner had a team of engineers and designers who worked on that area.  The consistency of people on the team was really helpful in driving creative ideas and effective output, because we all shared the same context of what our goals were, what we had tried in the past, and what we wanted to try next.  Netflix also has a great internal consumer insights team, and we would work with people from that team to use qualitative and quantitative research as appropriate to explore our ideas before launching them on the site or app.  We also had access to an analytics team to help us interpret the results (all the standard analysis was automated, but the team could help us drill into specific segments or non-standard metrics as needed).  

 

While I was there, international was also owned by each product owner, which let us localize the experiene as necessary but also apply key learnings globally (I'm not sure if that is or isn't still the case - there are good arguments to make pro and con of breaking it out separately, and we debated it vigorously).  

 

In general, I think the model of someone owning the core business as well as the testing and optimization is helpful.  Ideas can always come from anywhere in the organization, but it helps to have one person/team who has in-depth knowledge of the area and can fully flesh out a testing idea's implications.  

 

I also think it's great to encourage periodically revisiting core assumptions.  Over time, you tend to build up institutional knowledge, and sometimes forget the core assumptions underlying that knowledge.  When the assumptions change, you should re-test the institutional knowledge too.  Mobile is a great example of this -- early on, we found that people weren't interested in signing up on mobile, because the UI to enter information was too clunky.  As that UI got better, we revisited signup on mobile.

 

Best,

Jessie

 

 

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Joshua,

 

This is an interesting one, because the metrics you want to track are the ones that drive *your* business - so everyone's key metrics may be different.  

 

In general, I think marketing is responsible for two core things -- build a well-respected brand, and drive revenue/profitability.  The first is often hard to measure, but that doesn't mean it isn't important -- neglect your brand and you will usually pay for it later.  The second is easier to measure, and requires understanding your business.  For example, most media companies care a lot about page views, engagement, time on site/app, etc -- because they sell ad impressions.  The longer someone stays, the more ads they can sell.  Here at Optimizely, our business model is selling great software on a subscription basis to enable customers to build their businesses.  Marketing's role is to drive awareness of and interest in our product, as efficiently as possible.  

 

So for marketing effectiveness, we look at things like how many people visit our site, how many people sign up for a Starter account, how many people request to talk to our sales team, and ultimately how many people sign up for an Enterprise account.  We also care deeply about how much they use the product - how much of their site/app traffic runs through Optimizely? how many different experiences are they trying out?  what types of targeting are they using?  which integrations are they using?  All of that helps us understand how much value they are getting from using Optimizely, which helps us predict how likely they are to renew and to be enthusiastic about our product with others.  And of course, we look at how much we spend to drive that awareness and interest, so that we can track CAC (Cost to Acquire a Customer) relative to the projected customer lifetime value.  

 

Hope that helps!

Jessie

 

 

 

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Allan,

 

That's a really interesting one.  Yes, I've run into it sometimes.  

 

For me, I've always found it helpful to step back and look for common ground with the other team.  Usually everyone wants to do the right thing, and they're nervous that the other team is going to do things in a way that conflicts with their vision.  Build trust and relationships first, then you can get back to the changes you want to make.  If you can get both teams agreeing on a shared goal, that's a good foundation to explore.  Sometimes it helps to carve out a small area that is big enough for you to show what optimization can do, yet small enough that the other team is willing to allow some experimentation.  Once you can show results, and they trust that you have shared goals, you can usually expand to broader projects.

 

It also can help if you have a shared stakeholder, and that person can offer support for what you are trying to accomplish.

 

Getting companies to explore new ways of doing things can be hard.  I'd love to hear your ideas of other ways you've tried this and what worked and what didn't.

 

best,

Jessie

 

 

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Paul, 

 

This is a question we hear all the time, thanks for asking.  And, our answer to it has changed recently with the launch of Stats Engine, a way to read the results of your test without first picking a sample size.

 

There's really no "right" minimum number of visitors to a test. The minimum will vary for every business, and likely each type of page, and depends on a few factors: your preference for speed vs. risk and your typical traffic patterns.
 
In the past, we would have said to use the sample size calculator to see exactly how many visitors you would need to run to each variation. Now with Stats Engine, you can just set the test, and Optimizely will tell you when it's significant. Stats Engine is designed to detect eventually any effect that exists--but that also means you'll want to have an idea of when to stop your test if you're not seeing a significant result so you can move on to the next test.  The great thing about Stats Engine is you don't have to worry about peeking too early and seeing results that won't hold up over time - that was a big problem with the old way of presenting results.
 
You can still use our sample size calculator to give you a ballpark of how many visitors it might take to see the size difference you want to measure.  Calculating a sample size is effectively a reflection of how long you are willing to wait for your test to show significant results. The number of visitors you'll want to plan for depends on the baseline conversion rate and minimum detectable effect you're willing to wait for. The smaller the effect you want to see, the longer your test will likely need to run. 
We recommend inputting the smallest effect you're willing to wait for into the sample size calculator. If Stats Engine finds a larger effect, you'll know about it earlier--in some cases, up to twice as fast. 
 
You can also find results faster if you set your significance threshold lower than Optimizely's default 90% (this can be changed in your Optimizely project settings). A test run at 80% significance may find more false positives, but will also reach significance faster.  It's all about whether you prefer speed (with higher risk of false positives) or accuracy (takes longer, but lower risk of false positives).
 
The one other factor to consider for the minimum visitors to a test is your standard business cycle. For example, you probably wouldn't want to make generalizations about all of your visitors every day of the week based only on testing data from a weekend. 
 
Here are some other places you can read more about statistics, test results, and our Stats Engine
 
Let me know if this answers your question!
thanks
Jessie
 
--
Chief Marketing Officer at Optimizely
Optimizely
pcuci 02-11-15
 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Thanks Jessie - super helpful! I didn't know about the tools you mentioned which seem to address exactly the issue I described. Thanks so much,

Paul Cuciureanu
http://Enfluence.io - Digital High Blood Pressure Program
Join a team, get coached, and lead a healthier life!
Level 1

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi Jessie,  

 

I am looking for someone to optimize our website. In a way so that multiple pages are created similar to our main page, but we would like to be able to change the texts, pictures, and general theme of the page to better fit another market that we are targetting and also make the pages accessible from the main page.

 

Is that something you could help me with? 

 

Thank you

 

 

Brandon Nader

 

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi  Martin @MartijnSch

 

Interesting question!  Bottom line, I aim to spend my time where I think I can have the most impact for the company.  It's a combination of all the things you mention, and the amount of time I spend in each place varies depending on what's going on.  I think it's healthy to spend time with new people on a regular basis - I find inspiration in unexpected places, and it gives me new perspective to bring back to my "regular" work.

  

At a prior company, I had a colleague whose philosophy on allocating time was "Make a list of the top 3 things you could do today that will postiively impact the company.  Do those, then go home."  Realistically, many of us have lists longer than 3 items, but the point was, prioritize the most impactful things and get them done well.  That's what I aim for.  

 

How about you?  How do you allocate your time?

Jessie

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi @bd 

 

I think you're on the right path - you want to measure what is the relative return and cost of providing the discount to different audiences.  The key question is how much would a promotion change the response rate of repeat visitors, and is the incremental lift worth the cost of applying the promotion to all repeat visitors.

 

I've seen results go both directions -- in some situations, the lift wasn't nearly enough to justify giving the promotion to the full audience, while in others, the response lift was high enough to more than cover that cost.  

 

the one thing to be aware of is brand impact, and only you can make that judgment.  If people know that new customers get discounts but they don't, how does that make them feel about your brand?  an extreme example is some cable companies - they are notorious at offering amazing deals to new customers, leaving existing customers feeling taken advantage of.  In a world with more and more transparency, all your customers are likely to know about discounts available to some.

 

Good luck!  I'd be interested to hear what you learn.

Jessie

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi @BrandonNader 

 

If I'm understanding your question correctly, no, I don't think I can help much.  Sounds like you want to create brand new pages and update your navigation, yes?  

 

Jessie

--
Chief Marketing Officer at Optimizely
Optimizely

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

[ Edited ]

Well I am hearing that you are able to somehow provide different experiences for our customers on a single page. I.e. a customer will see a different message on our site depending on what they are interested in.

 

An example would be someone who searched for a pet related product, clicked on our link, will they be able to have that pet related experience? At the same time another person will search for a home related product, clicked our link, and will have a pet related experience. This all happens on the same page like a multi-website tool in one. 

 

Can you tell me more about personalization?

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Thanks Jessie, It's interesting to see someone else's perspective on data driven marketing.
------
Josh always rolling with the changes

"You must be shapeless, formless, like water. When you pour water in a cup, it becomes the cup. When you pour water in a bottle, it becomes the bottle. When you pour water in a teapot, it becomes the teapot. Water can drip and it can crash. Become like water my friend.”

Re: Ask Me Anything: I'm Jessie Becker, Chief Marketing Officer at Optimizely.

Hi @BrandonNader 

 

Sounds like maybe you're trying to do something like this?

https://help.optimizely.com/hc/en-us/articles/201231534-Optimizing-based-on-SEM-Google-Adwords-and-q...

 

Hope that helps,

Jessie

--
Chief Marketing Officer at Optimizely
Optimizely