Eliminate order bias to improve your survey responses

Why do stores design their layouts a certain way? Why do likeable and trustworthy people seem to mirror others’ behavior? And why do people use positive self-talk to improve their self-esteem?

The answers to each of these questions are the same: to influence people at a subconscious level to either believe or behave in a way that you deem desirable.

In survey design, it’s actually the survey creator’s job to avoid influencing respondents and instead, build a survey that elicits honest, thoughtful responses. Anything less, and the response data won’t be as accurate and as valuable as they need it to be.

To prevent bias in your survey, we’ll walk through common ways to limit your influence on the respondent’s subconscious.

Let’s start with a common source of survey bias, the order of answer options in a question.

What’s the most commonly used question type?

Multiple choice questions.

Since multiple choice questions are built around answer options, it’s worth getting comfortable with spotting and preventing answer option order bias.

This type of bias occurs when the order of your answer options influences the respondent to select a particular answer or a combination of answers. You see it most commonly in two forms:

How do you solve these survey biases? The best way is by randomizing the answer options of your questions.

To illustrate the impact randomization has on answer option order bias, the SurveyMonkey Research team conducted an experiment. We asked 400 respondents about their biggest pet peeve at work. Half of the respondents saw questions with randomized answer options while the other half didn’t.

Note: The order of answer options in the chart above is the same as the original question when its options weren’t randomized. The blue bars display the results when the questions weren’t randomized, while the green bars represent the results when they were randomized.

As it turns out, the results differed significantly between the 2 groups. Without randomization, the choices above the dashed line were more likely to be picked—giving us biased and misleading response data.

What about when the answer options were randomized? Each one had an equal chance of being selected. This led “Rude coworkers” to go from the second biggest pet peeve to the first by leaps and bounds.

Implementing answer option randomization on your next survey will be easy. Simply follow the steps in this article.  

Pro tips:

This survey bias comes in two forms. One relates to the relationship between the order of your questions and their level of specificity. The other revolves around bias inherent to questions that aren’t randomized—specifically when testing an ad or concept.

When asking the respondent about their experience on a given topic, you’ll almost always have one question about their overall experience followed by a series of more specific questions.

Your questions should flow from general to specific.

Why does the order of your questions matter? Because if you start by asking specific questions, it’ll lead to response bias—and potentially uncomfortable respondents.

For example, in this experiment, respondents were asked to report on their happiness in both their life and in their marriage. When asked to rate their general life happiness first, there was a small correlation between the answers and the questions. However, when they reversed the order of the two questions, they found that happily married people reported much higher life satisfaction. Since asking the marriage happiness question first influenced the respondents’ answers to the next question, the response data on general life happiness became biased.

Check out our customer satisfaction survey template. It can help you visualize the order of your questions as they go from a broad to an increasingly specific level.

Another common situation where question order bias arises is in concept testing—the process of refining a concept (ad campaign, product concept, new logo, etc.) by collecting feedback from your target market.

Let’s take a look at three more common tactics for addressing question order bias in concept testing: by randomizing the order of questions, page, and blocks (several consecutive related survey pages).

1) Question Randomization - When concept testing takes the form of images or videos, you’ll want to add each video or image to your survey and then ask respondents questions about them to determine which is the best.

Suppose you’re testing two ads. If you show them in a consistent order to every respondent, you’ll always have the first ad influence people's opinions about the second—but not the other way around. This biases your comparison between the two ads.

If you instead randomize the order each ad appears in, you’ll be left with a true comparison. Since the order respondents see the ad in changes, you can effectively “randomize out” the bias.

Here’s an example of an ad-testing template that uses question randomization. Take the survey multiple times to see the order of each image change.

2) Page Randomization - Instead of showing all the ads together, you can add one ad to each survey page with the corresponding follow-up questions on each page.

After that, you can randomize the order of your pages. After the responses come back, you can compare each image/video’s relative popularity by looking at the overall results.

Check out this survey. It puts each ad on a separate page. Like the survey on question randomization, you should take it multiple times to see how it works. This time, you’ll notice the order of the pages change.

3) Block Randomization - In case you’re testing an image or video with more than 1 page of questions, you’ll need to use block randomization.

To use block randomization, group your pages into blocks. Then choose to either show all of your blocks and randomize their order or show a single block at random to each respondent.

Here’s an image that shows how block randomization works in practice:

Also, check out this survey. It uses two ads and asks the same set of follow-up questions per ad. We’ve defined each ad and its two pages as a block, giving us two blocks in total. Each time you take our survey, you’ll view a single block. If you click on the survey link multiple times, you’ll see that a block is shown at random each time.

Note: Showing a single block at random may be the better option. As a single block only shows one ad, you’re removing question order bias. At the same time, you’ll still be able to aggregate the response data and compare both ads.

Survey design is filled with nuance. Whether it’s a survey’s block, page, question, or answer choice, any and all of these areas can lead to different types of bias. So try out the tips in this article to prevent response bias. Once you see the survey responses come in, you’ll be happy you took the right precautions.