Rich-text Reply

How does design influence your testing process?

ShanaR 10-13-15

How does design influence your testing process?

Hi Optiverse members,

 

I've been thinking about how design and testing/optimizaiton fit together. Are designers and optimizers fast friends, or roles with a healthy tension between them?

 

Do you use design to inform your test variations, hypotheses, or testing strategy? Does anyone have designers on their testing teams?

 

Curious to hear what you're already doing with design and optimization, what you'd like to learn more about, and any questions you have!

Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely

Alhan 10-13-15
 

Re: How does design influence your testing process?

[ Edited ]

Hi @ShanaR

 

Do we ever have designers on our team. They won't leave us strategists alone one second to make so-called "terrible design decisions". Seriously though, our designers play an important role in test planning. 

 

First off, it's important to distinguish between "designers" and "designers with years of UX design experience". The latter is the type of designer who will be really useful when it comes to the purposes of experience optimization. 

 

Let's say I have a problem I'm hoping to solve with an upcoming A/B test. The problem might encompass any number of the following elements: copy, functionality, imagery, information architecture, etc. Rather than focusing on being overly specific in my proposed solutions, I will provide all of the evidence and hypotheses to a designer. The designer will then come back with a number of possible solutions. We end up coming up with solutions by using everyone's strengths.

 

While strategists are good at understanding the customer, UX designers are good at turning that understanding into a better experience. 

 

What's also nice about the process is that a designer will take into consideration the full user journey from a UX perspective rather than looking at the test in a vacuum, which is important when you're doing a lot of testing up and down your conversion funnel. 

Alhan Keser, Optimization Strategist @ WiderFunnel

Product Manager of Liftmap - Plan, track and share your Optimizely experiments with anyone.

Optimizely Platform Certified
Level 2
ShanaR 10-14-15
 

Re: How does design influence your testing process?

@Alhan, thanks for the response and for sharing your workflow. Really interesting process and lots of added benefits to having a UX designer think about the best way to test through a series of ongoing changes.
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
MartijnSch 10-14-15
 

Re: How does design influence your testing process?

Hi Shana,

Great question, in the last 2 months we've had our designers way more involved into our process of testing. We found that the longer we test the less ideas we could come up with ourselves and the more basic they were in terms of design. As our designers are of course working 24/7 on the design they can come up with more crazy ideas but also know from time to time on how to execute them.

As they're working on some redesigns it allows us to work directly with them on implementing their ideas to proof these concepts.

I'd love to have designers dedicated on our testing team but I don't see that happen in the next 6-12 months as their support currently is already of great help in the things that we're doing.

Hope this helps!
scleveland 10-14-15
 

Re: How does design influence your testing process?

Hi Shana

 

We are a smaller company, but currently our team is composed of an Analyst, Developer and a Designer. 

 

Each week we meet about our sites.  Come up with what we want to test.  Hand that over to our designer to come up with mockups of what she thinks the test should look like.  Then when we've agreed on the design, she turns it over to our Developer who sets up the test and ultimately runs the test.

 

So to answer your question, our process is literally nothing without our designer.  Plain and simple. 

 
 
 
 
 
c5wales 10-14-15
 

Re: How does design influence your testing process?

HI @ShanaR,

 

We sit underneath IT and the umbrella of IT covers all things web. Underneath this unbrella we have Analytics and Optimization and UX. Typically, analytics will create a hypothesis based off data we see in our analytics tools and UX will help guide us in the design of the test. Then we'll have our developers build the test out to UX specifications. It has been difficult to have UX/Design play a bigger role in our testing process since the team was comprised of one UX Director. We now realize the importance of us working togeher and are bringing more UX designers on to create more bandwith.

Chad Wales
Product Development Manager
American Medical Association
Level 2
JeremyBrad 10-14-15
 

Re: How does design influence your testing process?

Hello,

I've enjoyed reading these responses and seeing how other teams are joining forces with their design and UX departments to come up with new test ideas. In my experience, there has always been a natural tension between those who've traditionally been granted expert status, and those who just want to test everything while presuming to know very little. Prior to having a strong emphasis on a/b testing in our company, it was assumed that what design created was best simply because they know best. And what was best for one page, was probably best for all pages. As an Optimizer, I tend to focus nearly exclusively on what's best for conversion, whereas design tends to put more emphasis on aesthetics and site consistency. Clearly these are all important goals, but they are sometimes at odds and resolving that conflict has been challenging. I suspect, as with most conflicts, good and open communication is the best path forward. It sounds like a lot of folks have had success by involving design in the earliest phases of constructing a test so that they don't feel as threatened by attempts to improve upon their prior work. Great food for thought - Thanks.
Jeremy Bradford
ShanaR 10-14-15
 

Re: How does design influence your testing process?

@MartijnSch Very interesting that this is a more recent workflow for your testing program - thanks for sharing! Glad that the impact has been net positive.
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-14-15
 

Re: How does design influence your testing process?

@scleveland That's a pretty powerful combination of skills - thanks for outlining your workflow and sharing with us!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-14-15
 

Re: How does design influence your testing process?

@c5wales Thanks for sharing your team structure - bandwidth is always a challenge when you're trying to move quickly and test quite a bit. Also interesting that you, like a lot of Optiverse members, are looking for design to play a generative role in creating test variations. Thanks for your response, and keep us posted on how things evolve!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-14-15
 

Re: How does design influence your testing process?

@JeremyBrad Interesting observations and insights. I think that proactively organizing testing teams to include designers is an important step towards making the two disciplines work together, but not a luxury that everyone has.

I also think you're dead-on in your assessment of conversion focus vs. aesthetic focus, and how those two priorities can sometimes conflict.

Hopefully, educating both audiences on how they can work better together will lead to a more beautiful and high-converting final product! That's what I'm interested in digging into. Thanks again for sharing!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely

Re: How does design influence your testing process?

I'll agree with Jeremy on the difference between aesthetic design vs. conversion-focused design. We are at a point with our optimization team where we have dedicated design resources, which makes us not only more nimble but able to put together designs that we feel are the most successful for whatever our success metric is. We've had plenty of experience in the past with designers from other parts of the organization who, while talented, don't really understand conversion. It's tough when we have to work with teams who are very set on the 'style guide,' and we are sort of seen as rogue agents sometimes because we deviate. It's a continual struggle but a very important one!

Renée Thompson
JeremyBrad 10-14-15
 

Re: How does design influence your testing process?

@ReneeThompson It's nice to hear that others are confronting the same difficulties. I can really identify with being perceived as "rogue", since it's basically our job to question the status quo. It's at least half the fun and half the difficulty of doing optimization work. Of course I'd rather be rogue than a conformist, but it does come with it's own set of difficulties. Luckily, the more success we have with a/b testing, the more that perception seems to be going away.
Jeremy Bradford
tscala 10-15-15
 

Re: How does design influence your testing process?

Hello Shana,

 

Design absolutely informs Hanley Wood's test variations. Our Design/UX/UI team is an integrated memeber of our testing team, and while we use testing to stretch boundaries and freely experiment, we always do so within the aesthetic and experience guidelines of our brands.

 

Thomas Scala

Level 2
glvzLIFT 10-15-15
 

Re: How does design influence your testing process?

Awesome discussion @ShanaR!

 

Across companies I've worked with, I've seen more success with there being a healthy tension. When test ideas and variations are vetted with a critical eye from both design & optimization, it raises the overall quality and potential impact you can make.

 

The influence of design typically ramps up as we move closer to test execution. As much as designers live in the world of ideas and problem-solving, they're not always that interested in following the data. I've often seen a disconnect between what designers want to do and what site visitors respond to. I've found some success in framing insights from data as design challenges.

 

Having worked with both structures, and there are definitely benefits to having them on-the-team. At the very least, it can help with getting test-related requests prioritized and resourced. That said, it's not a guarantee that there will be strong collaboration between the strategist and the designer. In the end, the collaboration can be nurtured across any reporting structure, and it's crucial.

 

Michael Galvez

Optimization Strategist @Pointmarc

Michael Galvez
Optimization Strategist, Pointmarc
Level 2
ShanaR 10-15-15
 

Re: How does design influence your testing process?

@ReneeThompson Interesting history - thanks for sharing!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-15-15
 

Re: How does design influence your testing process?

@tscala Sage advice, thanks for sharing your perspective!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-15-15
 

Re: How does design influence your testing process?

@glvzLIFT Thank you for joining in! Appreciate the additional context on challenges and how the workflow of a joint design+optimization team provides added benefits.

Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-19-15
 

Re: How does design influence your testing process?

Hello everyone!

Since this is clearly a rich topic of discussion, we're piloting a new series on the Optimizely blog: Design Principles You Should A/B Test. Check out the first one here https://blog.optimizely.com/2015/10/19/design-principles-you-should-test-color-contrast/.

@Alhan, your great quote is featured.

Thanks to everyone who participated in this thread, and let me know if there are any topics related to design + optimization that we should dig into in the future!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
Alhan 10-19-15
 

Re: How does design influence your testing process?

Thanks, Shana!
Alhan Keser, Optimization Strategist @ WiderFunnel

Product Manager of Liftmap - Plan, track and share your Optimizely experiments with anyone.

Optimizely Platform Certified
Level 2
juliofarfan 10-19-15
 

Re: How does design influence your testing process?

I need a lot of design in the tests im running however Design team is not highly involved due to response time. So we mostly do the design work on the optimization team.
Amanda 10-19-15
 

Re: How does design influence your testing process?

[ Edited ]

@juliofarfan - Is there a specific tool that you use that allows your optimization team to do the design work? I know someone from our team loves Sketch since it's more manageable than something like Photoshop. I'm curious how you enable your team to take on the design work (and I'm sure this would be a valuable piece of information to others as well!)

Optimizely
juliofarfan 10-19-15
 

Re: How does design influence your testing process?

We mostly use Photoshop.

Also, there are certain conditions when a machine with Photoshop is not available so we use Paint.net recolor tool to create images already existing but with different colors. O even in some cases Paint.
jfx1026 10-19-15
 

Re: How does design influence your testing process?

Design is an integral part of most of our tests. Whenever we create a test the brand guidelines and effect on the overall user experience are always considered very carefully.

 

It's rare that a test does not touch design in some way.

http://johnfreeborn.com
Level 3
Amanda 10-20-15
 

Re: How does design influence your testing process?

@jfx1026 - Thanks for sharing! How do you make sure that your tests are inline with brand guidelines? Do you leverage designers from within your organization? Or do you enable people on your own team to be educated enough on the guidelines to do their own design?

Optimizely
EH 10-21-15
 

Re: How does design influence your testing process?

Design is critical to our success so we implement it right away. Our designers most likely will pull something out that they worked on before that benefits our current project and/or will help us keep our design standardized across different areas of our site to keep the user experience consistent.
EH
Level 2
robertchan 10-26-15
 

Re: How does design influence your testing process?

Designers and people working in optimization should work together with a sense of balance. At my company, I meet with our designer on a weekly basis to make sure we are on the same page in terms of modifications I've made. However, priority will sometimes override our designer's decision based on how fast an experiment needs to be released.

Robert Chan

Experimentation Hero
ishaan 10-27-15
 

Re: How does design influence your testing process?

Hi ShanaR,

I feel that designers and optimizers are fast friends, and go hand in hand. You can not optimize without knowing the design restrictions. You should work in a team for delivering better experience.

 

Our team does have a desgner. We take the feedback for each testing opportunities and based on the UX design, we built our test. its an iterative process. First, we build the test and show it to the design people. They provide feedbcak and we then do the changes and once again review it with the design team. When we are on the same page, we go ahead and launch the test.

 

Thanks,

Ishaan.

Level 2
ShanaR 10-27-15
 

Re: How does design influence your testing process?

@EH Thanks for sharing!
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely
ShanaR 10-27-15
 

Re: How does design influence your testing process?

@robertchan Great perspective, thanks for sharing your tips! It sounds like communication and a shared understanding of priorities are key to a great design and optimization relationship.
Shana Rusonis
Content Marketing Manager, Optimizely
shana@optimizely.com
Optimizely