Surprising Test Result: Reducing the number of form fields did not increase conversions

[#1]

The conversion page I wanted to test was a form where the user had to enter in their personal information in order for a dealer to contact them. (It's a lead gen site) Well, my obvious assumption was that less form fields = greater conversions but suprisingly that wasn't the case. 

Hypothesis: Reducing the number of form fields will increase conversions 

description: remove all form fields but name, phone and email

The variation almost never outperformed the orgiinal outside of the initial testing timeframe. It was a little disheartening not being able to increase conversions but I came to realize that for certain products, users expect to provide a lot of information. It's like only being asked for a name and email to apply for a credit card. It just doesn't feel right.

I believe I've encountered a similar situation. More testing is needed

[#2]

@trangm This is very interesting, thanks for posting! Can you tell us a little more about the type of service or product that your site sells? 

Your finding reminds me of an article I recently read about how Lumosity tested complexity vs. simplicity in their sign up flow. The found the following: As they made registration more complex, users actually became more valuable.

Here's a quote from the article:

“What we found is that sometimes friction can help you acquire customers that really believe in your product, who want to build a long-term relationship with your company”

You can check out the article here: http://firstround.com/article/How-Lumosity-Spiked-Active-Users-10-with-Complexity-Not-Simplicity

[#3]

@trangm - Is this a screenshot of your test results? My initial concern would be that there is no significance in your numbers which means that there could actually have been a chance that this test would have produced a positive uplift?

How long was the test run over?

I would like to point out that I am not trying to shoot down your test. Just want to make sure you are working with the correct information as it could affect future A/B testing decisions!

[#4]

@Amanda  We sell HVAC equipment which can cost upwards of $20k. Also, thanks for the link. I'll definitely check it out.

@adzeds yes, this is a screenshot of my test results. There was no significance in my numbers which led me to the conclusion that the new variation had no affect. The test was run for 2 months. And you don't have to point out anything. Any constructive criticism is appreciated. 

[#5]

I think the question raised by @Amanda reveals a great point about lead generation forms - the higher specificity or complexity of the product or information being requested may necessitate MORE customer information up front in order to generate more qualified leads.  In your case, purchasing large-scale HVAC equipment is going to require a thorough discussion with the potential customer and evaluation of their business needs.  The customer knows and expects they need to provide a good amount of information to you in order to facilitate a purchasing conversation.  They don't want their second contact with you (i.e. the phone call following up on their form submission) to just be about collecting information they could have already provided.

In contrast, think about something like Facebook.  You can spend days and weeks filling out a complete profile, but if you had to do that just to sign up and get started, no one would ever join.  That's a site where a piecemeal sign-up form makes more sense.

Thanks @trangm for highlighting this!

[#6]

My first thought was along the same lines as what @adzeds  mentioned. Your screenshot only shows around 70 conversions per variation which is rather low, even though this ran for two months. As a general rule within our team we try to make sure every variation receives at minimum 100 conversions. We've seen drastic swings in conversion rates between when variations only have 50-60 conversions to when they have 150+ conversions. It's possible that with more conversions your results might change. Just a thought.

But what others suggest also makes perfect sense: that a more complex product requires more complex information so users are comfortable with more form fields. I'm curious to know how many and which fields you removed. Do you mind sharing that?

[#7]

Thanks for the feedback @Brian_DeLoach. After hearing what you and @adzeds had to say about my experiment, I'm going to open it back up and wait for more conversions(150?). I'm interested to see if there will be any drastic swings in my numbers.

I'll report my findings on this comment thread. 

Cheers,

Trang