Ask Me Anything: I’m Dan Siroker, Co-Founder and CEO at Optimizely

by Optimizely ‎05-09-2016 May 9, 2016 - last edited on ‎05-12-2016 May 12, 2016 by Optimizely

Dan1 copy.jpgThe live Q&A portion of this AMA is closed. Scroll below to see all of the answers.

 

Hello Optiverse! I’m Dan Siroker, the Co-Founder and CEO of Optimizely. Through our A/B Testing and Personalization products, we empower companies to optimize their websites and mobile apps and turn data into action. Ask me anything.


A little bit about me: The original inspiration for Optimizely came from my experience as the Director of Analytics on the 2008 Obama campaign. In 2010, my co-founder Pete Koomen and I founded Optimizely - the product I wish I had in 2008 to make it easy for anyone to do A/B testing. I was formerly an APM at Google and graduated from Stanford in 2005 with a degree in Computer Science.

 

I'm looking forward to answering your questions!

 

--

If you are interested in being featured as an Expert for a specific topic, please email optiverse@optimizely.com

Comments
by Mia
‎05-09-2016 May 9, 2016

Hi Dan! What is one of the most interesting or unique ways you have seen a customer leverage Optimizely? 

Mia
Level 3
by SBarbera
‎05-09-2016 May 9, 2016

Hi Dan,

 

Are there places and situations where you would recommend that Optimizely, not be used?  Could you provide a few examples?

Level 1
by parththakkar
‎05-09-2016 May 9, 2016 - edited ‎05-09-2016 May 9, 2016

A product like optimizely is a massive engineering feat. Were you able to do some of the MVP dev yourself or did you enlist a developer? If you did get a developer to join on, how did you go about looking for one?

Bonus: What is your favorite breakfast item? Smiley Happy

Level 5
by
‎05-10-2016 May 10, 2016

Hi Dan! What do the best optimization teams in the world do different?

Level 10
by
‎05-10-2016 May 10, 2016

Hi Dan! What are some new products we can expect from Optimizely in the future?

Experimentation Hero
by leepearso
‎05-10-2016 May 10, 2016

What tools do you think compliment optimizely?

 

Which competitors do you think do a great job and why?

Level 2
by randyfong
‎05-10-2016 May 10, 2016

How popular is Optimizely used on the two mobile platforms? Is it used more frequently on IOS or Android?

Level 1
by mobe
‎05-11-2016 May 11, 2016

Whats your opinion on multi-arm-bandit algo used by Google Analytics Content Experiments to reduce Experiments runtime.

Level 2
by parththakkar
‎05-12-2016 May 12, 2016

Hey Dan, how would you characterize the culture of the workplace of you first job? Did it have an impact on the culture that you promote at Optimizely? 

Level 5
by ilincev
‎05-12-2016 May 12, 2016

Hi Dan,

are you considering changing the pricing? Adding more tiers. Now you can either choose a free version or Enterprise, which starts around $400. SMB clients usually budge and go for VWO instead.

Level 3
by AlexisI
‎05-12-2016 May 12, 2016

Hi Dan,

Any plans to add another level or to to Folders. Would be great to be able to organize underneath the Project Level.

Level 2
by rob775
‎05-12-2016 May 12, 2016

Hi Dan

Congrats on all your success! 

 

Perhaps perception isn't reality, but working with 3rd party data sources seemed to be a priority for Optimizely in the past, but so much as of recently. Do

Do you think Optimizely will prioritize making external audience integrations available in the future? 

 

Thanks!

Level 1
by anderson
‎05-12-2016 May 12, 2016

Hi Dan, do you consider generate a program for non-profit organizations ? I am ifantiles Villages and use the tool

Level 1
by gmorgan
‎05-12-2016 May 12, 2016

What's the most creative and effective application of Optimizely that you've ever seen?

Level 1
by angilly
‎05-12-2016 May 12, 2016

Hey Dan,

 

Ryan from Ramen (https://ramen.is) here. Can you walk us through how you got customers 10 through 100? I've found that getting the first few customers is actually relatively easy, but it's getting from 10 to 100 that kills most companies. At that stage, were you doing lots of 1-on-1 meetings, sitting behing a screen executing marketing campaigns, leveraging PR, or something else cool?

 

Thanks!

 

--

Ryan Angilly

CEO Ramen

Level 1
by EmilyPierce
‎05-12-2016 May 12, 2016
Hey Dan. This is a pretty broad question. Do you have any advice for establishing a baseline conversion rate for a given industry?
Level 1
by Littlelizzyv
‎05-12-2016 May 12, 2016
Hi Dan,

With the onset of virtual reality becoming more accessible, how does Optimizely plan to utilize the new technology to make a better user experience?

Thanks!
Level 1
by robotoole
‎05-12-2016 May 12, 2016

Hi Dan,

 

Do you know how badly performant your android sdk is? It is the primary cause of concern and lag for the startup of ours and all android applications implementing your sdk. What do you plan to do to fix that? I've attached some output from testing of the startup of our app in a great tool at nimbledroid.com. As you can see, the main culprit is you.

 

NimbleDroid___Automated__Comprehensive_Performance_Analysis_for_Every_Build_of_Your_App.png

 

 

Level 1
by brgormz
‎05-12-2016 May 12, 2016

What's the most interesting test you've seen an Optimizely customer run (that you can share)?

Level 1
by xx
‎05-12-2016 May 12, 2016

Dan, when is the next Optimizely meetup with tacos and beer on Howard St? 

 

Howard St.

xx
Level 1
by tamireiss
‎05-12-2016 May 12, 2016

What was it like working on the Obama campaigns?

Level 1
by CD1
‎05-12-2016 May 12, 2016

Does Optimizely have a feature for data cleansing to identify/normalize outliers? 

CD1
Level 1
by roh
‎05-12-2016 May 12, 2016

Can A/B testing and Personalization hurt your existing SEO?  If we ran an A/B test, will Google try to index both versions? 

roh
Level 1
by roh
‎05-12-2016 May 12, 2016

One small recommendation: might recommend that you move your input box to the top of the page (just above the first comment/question), as it took me a bit of time to figure out how to submit a question...might be worth a test.  Thanks.

roh
Level 1
by Peter26
‎05-12-2016 May 12, 2016

Why the incredibly expansive free version?  (I mean don't get me wrong, I love it, but curious about the thinking behind providing so much for free.)

Level 1
by cledford
‎05-12-2016 May 12, 2016

What advice would you give founders who are working to grow from the 2m ARR level to 10m ARR.

Level 1
by MattCallegari
‎05-12-2016 May 12, 2016

Dan -

 

  1. If you were just starting Optimizely and all you had was an MVP version of your product and $1,000 - how would you spread the word? How would you "get it out there"?

  2. What is the best way to test a new business idea?

  3. What separates good ideas from great ideas? 

 

Thanks so much for everything!

 

- Matt

 

by Sarim
‎05-12-2016 May 12, 2016

Hey Dan,

A couple of questions...

 

Q1. What was your most successful growth hack during the initial days of optomizely?

 

Q2. Have you ever used reddit to get more web traffic? If yes, what strategy did you use?

 

Thanks,

 

Sarim

 

Level 1
by Optimizely
‎05-12-2016 May 12, 2016 - edited ‎05-12-2016 May 12, 2016

@Mia Hi Dan! What is one of the most interesting or unique ways you have seen a customer leverage Optimizely? 

@gmorgan What's the most creative and effective application of Optimizely that you've ever seen?

 

One of the most interesting ways I've seen customers use Optimizely is as a light-weight CMS on top of their existing website. After customers run an experiment and a new variation is declared the winner they commonly will go in and send 100% of traffic to that variation and keep it that way for a long time. This buys their team time to go back and hard-code the winner, or, in some cases, works perpetually as a way to deliver more optimized experiences directly through Optimizely!

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@SBarbera Are there places and situations where you would recommend that Optimizely, not be used?  Could you provide a few examples?

 

Optimizely only works on websites where you can add a line of JavaScript or mobile apps where you can install our SDK. An example of a thing Optimizely can't be used is making changes to flash or java components since neither of these can be manipulated directly with JavaScript.

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@parththakkar A product like optimizely is a massive engineering feat. Were you able to do some of the MVP dev yourself or did you enlist a developer? If you did get a developer to join on, how did you go about looking for one?

Bonus: What is your favorite breakfast item? Smiley Happy

 

Thanks! My co-founder, Pete Koomen, and I both studied computer science so we were able to build the original MVP for Optimizely on our own. Only after we built a product and had a few paying customers did we raise money and hire more engineers. It's much easier to hire engineers once you have proven traction and momentum. Smiley Happy

 

My favorite breakfast item is bacon! mmm, bacon.  Smiley Happy

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@MartijnSch Hi Dan! What do the best optimization teams in the world do different?

 

The best optimization teams focus on asking the right questions, not having the "right" answers. Their approach to optimization is a very humble, data-driven one, that is focused on being objective and avoiding bias. Sometimes the best ideas come from questioning things from first principles, not just doing the same thing you've always been doing. Hope that helps!

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@robertchan Hi Dan! What are some new products we can expect from Optimizely in the future?

 

We focus on deeply understanding our customer's biggest problems and building solutions they might not have ever imagined could exist. Today, we see many such opportunities. One in particular that is really excited to me is the opportunity to help businesses A/B test more than just their website. Going deeper into the tech stack and enabling developers to A/B test the functionality of their apps is a huge opportunity. I saw this first hand at Google and I want to democratize the kind of tools we leveraged back then to enable anyone in the world to create better digital experiences.

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@leepearso What tools do you think compliment optimizely? Which competitors do you think do a great job and why?

 

Our customers want to use best-of-breed products for everything they do and expect those products to integrate seamlessly with one another. That is why we deeply focus on A/B Testing and Personalization and seamlessly integrate with everything else. A great example of an integration that complements Optimizely well is Demandbase, which is something we use ourselves at Optimizely through our integration. Here is some more info on that: https://help.optimizely.com/Integrate_Other_Platforms/Integrating_Optimizely_with_Demandbase

 

Regarding competitors that do a great job, I think Adobe does a great job of marketing themselves. We benefit tremendously from all of their marketing because it helps lift up our entire industry. Smiley Happy

dan
Optimizely
by Joey_Herzberg
‎05-12-2016 May 12, 2016

Hi Dan, What is the best way to get developers and product owners to buy into A/B testing?

 

Thanks!

 

-Joey

by Optimizely
‎05-12-2016 May 12, 2016

@randyfong How popular is Optimizely used on the two mobile platforms? Is it used more frequently on IOS or Android?

 

Quite popular! According to mixrank.com we are the most-adopted A/B testing SDK. It is currently being used more frequently on iOS but we expect Android to catch up.

dan
Optimizely
by
‎05-12-2016 May 12, 2016

@dan - thanks for the response, Dan!

Experimentation Hero
by Optimizely
‎05-12-2016 May 12, 2016

@mobe Whats your opinion on multi-arm-bandit algo used by Google Analytics Content Experiments to reduce Experiments runtime.

 

We've experimented with this for a while and we've learned from our customer's is that they want to track many different goals simultaneously and the decision to allocate traffic to a winner isn't always as simple as choosing a primary metric and having the system do the rest. That said, we want to continue to give our customers more flexibility on how to automatically allocate traffic so stay tuned!

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@parththakkar Hey Dan, how would you characterize the culture of the workplace of you first job? Did it have an impact on the culture that you promote at Optimizely? 

 

My first job was at Google so it did have an impact on the culture that I promote at Optimizely. I took the things I loved about Google's culture, like transparency and trust, and brought that to our culture. There were a few things at Google that I didn't like, namely I never felt true ownership over what I worked on or had radical autonomy to do fearless things. These are two aspects I've incorporated into our culture here at Optimizely which makes Optimizely the kind of place I would want to work as an employee. Our culture is best described as living up to the following cultural values: Ownership, Passion, Trust, Integrity, Fearlessness, and transparencY (OPTIFY).

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016 - edited ‎05-12-2016 May 12, 2016

@ilincev are you considering changing the pricing? Adding more tiers. Now you can either choose a free version or Enterprise, which starts around $400. SMB clients usually budge and go for VWO instead.

@Peter26 Why the incredibly expansive free version?  (I mean don't get me wrong, I love it, but curious about the thinking behind providing so much for free.)

@Dreamboat27 For a while, Optimizely offered a freemium model, where the service was free for most small customers, and the only paying customers were Enterprise-Level.  Could you explain how you came to try that model, and also why you moved away from it? 

Thank you!  I love Optimizely!

 

 

We've learned a lot about pricing over the years. The most important thing I want to accomplish is align the price of our product to the value we deliver. I want to do this in a way that enables our customers to go on a journey with us from dipping their toes in to us truly becoming an essential partner to them. This philosophy has inspired our fully-featured Pay-As-You-Go pricing where you can get a $500 credit as a way to get started to enterprise pricing that enables you to pay less if you use less functionality. Most ex-VWO customers who try their product and come to Optimizely afterwards admit that the total cost of ownership is lower with Optimizely because we enable many more people in your organization to run tests and we don't tax them for the more tests they run. Most companies that are serious about A/B testing recognize this and appreciate our scalable pricing model.

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@airion Any plans to add another level or to to Folders. Would be great to be able to organize underneath the Project Level.

 

Thanks for the suggestion! It's on our roadmap. Smiley Happy

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@rob775 Congrats on all your success! Perhaps perception isn't reality, but working with 3rd party data sources seemed to be a priority for Optimizely in the past, but so much as of recently. Do you think Optimizely will prioritize making external audience integrations available in the future? 

 

As I said earlier, we aim to focus on A/B Testing and Personalization and seamlessly integrate with everything else. We currently have more integrations than any of the alternatives in the market. That said, if there are specific external audience integrations we don't already support that you'd like, let me know!

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016 - edited ‎05-12-2016 May 12, 2016

@anderson Hi Dan, do you consider generate a program for non-profit organizations ? I am ifantiles Villages and use the tool

 

Absolutely! Check out: http://optimizely.org

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@angilly Can you walk us through how you got customers 10 through 100? I've found that getting the first few customers is actually relatively easy, but it's getting from 10 to 100 that kills most companies. At that stage, were you doing lots of 1-on-1 meetings, sitting behing a screen executing marketing campaigns, leveraging PR, or something else cool?

 

The most effective & scalable solution is to turn your first 10 customers into evangalists! Our best sales team are our existing customers. The more you can do to invest in making those handful of initial customers wildly successful, the more likely they will spread the word. The other thing we did early on that helped a lot was speak at marketing conferences. I would share stories from the Obama campaign in 2008 and then at the end of my talks always make a brief pitch for Optimizely. For years afterwards people would mention they first heard our story at these conferences.

 

 

dan
Optimizely
by rob775
‎05-12-2016 May 12, 2016

Thanks @dan 

We are B2B and enjoyed using Bizo data. Adobe now has Bombora data so we are considering a swtich.. 

Level 1
by Dreamboat27
‎05-12-2016 May 12, 2016

Dan, 

For a while, Optimizely offered a freemium model, where the service was free for most small customers, and the only paying customers were Enterprise-Level.  Could you explain how you came to try that model, and also why you moved away from it? 

Thank you!  I love Optimizely!

Tim

Level 1
by dogdaydarren
‎05-12-2016 May 12, 2016
Why is traffic allocation for A/B tests in Optimizely often very far from 50% each? Why are you allocating traffic randomly instead of precisely rotating it between variants? Couldn't a random allocation risk involving time of day factors when very likely one variant will get more traffic at one time of day?
Level 1
by Optimizely
‎05-12-2016 May 12, 2016

@EmilyPierce Hey Dan. This is a pretty broad question. Do you have any advice for establishing a baseline conversion rate for a given industry?

 

This is a really tough question because a conversion means something very different in each industry (e.g. for retail it's usually a purchase, for B2B it's a lead, for media it's an ad click). There is some good stuff out there for just retail, though. Here is a great resource: http://www.marketingsherpa.com/article/chart/conversion-rates-retail-categories

 
dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@Littlelizzyv With the onset of virtual reality becoming more accessible, how does Optimizely plan to utilize the new technology to make a better user experience?

 

Our vision is to enable the world to turn data into action. Today, we are focused on the web and mobile apps but our ambition is to enable our customers to optimize all of the digital experiences they deliver to each of their customers, including virtual reality. Smiley Happy

dan
Optimizely
by Optimizely
‎05-12-2016 May 12, 2016

@robotoole Do you know how badly performant your android sdk is? It is the primary cause of concern and lag for the startup of ours and all android applications implementing your sdk. What do you plan to do to fix that? I've attached some output from testing of the startup of our app in a great tool at nimbledroid.com. As you can see, the main culprit is you.

 

Thanks for the candid feedback. I'd recommend you load Optimizely asynchronously using startOptimizelyAsync. The Android tutorial shows how to do this: http://developers.optimizely.com/android/guide/index.html

 

Hope that helps!

dan
Optimizely