Pop quiz!
Read this list of words quickly: pen, airplane, shell, butter, duct, blueberry, wrapper, zebra
Now cover the list with your hand.
Now list all of the words you remember.
If you’re like most people, you’ve remembered the first one (pen) and the last one (zebra), but not so many in between. How did we know that? Well, the order of a list can be just as important as the actual content. The order itself changes how you think and respond.
The same thing happens in surveys when you give respondents a list of answer choices. The first and the last answer choice stick out, and the middle tends to get ignored. Take this question for example:
More respondents will pick Harry Potter and Percy Jackson than would have otherwise simply because they are the first and last choices. In survey language, that’s called a response choice order effect.
Why might people pick the first response choices?
- Satisficing. Respondents start with the first option, thinking about all the reasons the Harry Potter series might or might not be the best series of the last ten years. They then do the same thing for the rest of the options on the list. As respondents go down the list, they start getting mentally tired. They then begin to satisfice–they stop paying close attention and skim over the last options in the list, neglecting to consider them carefully. This can leave the top response choices in the list with a much stronger case than the bottom ones.
- Priming. We talked before about how this works with questions. Ask someone their income and that’ll sway how they respond to a question asking if they believe in the American Dream. Believe it or not, the same thing happens with response options too. For example, if Harry Potter is first on the list, respondents might first think about how the series made billions of dollars. They’ll then use this same standard–which is pretty unique to Harry Potter–to judge the other options in the list. None of the other options made billions of dollars, so Harry Potter is left as the easy choice.
Why might people pick the last response choices?
- Satisficing. Yes, satisficing rears its ugly head here too. Again, respondents do the hard critical thinking work for the first few options, but get mentally tired and begin to satisfice. When they stop thinking critically about the last few choices, this can actually make them seem more appealing, not less. It becomes more of a gut instinct selection than a rationally thought-out opinion.
- Recency. Sometimes it’s just the last thing you hear that sticks in your mind. It’s not because you’re being sloppy or because you’re not thinking clearly but just because sometimes the last thing that you’re hearing seems the most compelling. Think of it as years of conditioning that you always “save the best for last.”
Does response order REALLY matter that much?
YES! For example, studies of voting have shown that the order of candidates’ names on a ballot affects vote totals. For example, in the 2000 presidential election, George W. Bush got 9 percent more votes when he was listed first on the ballot than when he was listed last–that’s a huge difference! This is one of the reasons that ballots randomize the order of candidates’ names.
Okay, then how do I fix it?
- You can randomize the options, so that each person sees the list in a different order.
- Sometimes, if you have an order list or a rating scale, randomizing the options can feel awkward. In this case, you can flip the options, so some of your respondents see a list that begins with Harry Potter and ends with Percy Jackson, and others see a list that begins with Percy Jackson and ends with Harry Potter.
If you randomize or flip the options, any order effects will balance themselves out and your data quality will be protected! And guess what? Both of these solutions are just a click away with SurveyMonkey.
Be sure to let us know how it goes in the Comments section below!
