The order questions appear in your survey can directly impact the responses you gather. One of the more well known examples (Iyengar & Kinder, 1987) of question order effects is in the domain of politics. When pollsters ask people, “What is the most important problem facing the nation?,” the answer they give becomes the object of focus for their answer to a subsequent question—“Do you approve or disapprove of the way [insert name] is handling his job as president?” By and large people answer the approval question while judging the president primarily on his performance on the issue they consider to be most important.
This phenomenon is called priming. Respondents are primed to think about one issue while answering the subsequent question.
Priming has an impact even if the survey topic is not political, or even controversial. Let’s take a seemingly harmless survey. Let’s say you're trying to gauge sentiment for a company softball team. You may be tempted to ask “What is your favorite sport?” as part of your survey. Seems like a fair start. But if your next question is, “How interested are you in playing on a company softball team?” and softball isn’t high on a respondent's list of favorite sports, then they may be tempted to rate their interest lower in joining the company team than if you had asked them first about the team, then about their favorite sport.
Another reason question order matters is that respondents may have a desire to appear consistent in their responses. For example, if you ask students to answer a very difficult math problem first, then ask how much they enjoy math, they may be tempted to rate their interest lower if they struggled to solve the math problem.
Even response options from an earlier question can impact subsequent answers. If you ask “Which of these four fruits is your favorite?” and then ask how much fruit they eat in a week, the respondent may focus their attention on just those four items and report a lower number than if they were to think of a longer list of fruit.
How can you address question order effects in your own surveys? One option is to randomize your questions so that respondents are not all answering questions in the same order. You can learn how to implement Question Randomization by visiting this help center article.
