When it comes to surveys, there’s practically nothing worse than satisficing.
Satisficing is a fancy word for when people don’t put as much effort into answering your survey as they should. That might not sound like a big deal, but it can really affect your data.
It’s easy to think of satisficing as a problem with respondents, since they’re the ones who aren’t trying hard enough to give thoughtful answers on your survey. But as the survey designer, the burden falls on you to create a survey that makes it easy for people to give their opinion.
The truth is that surveys run on good will. Think of your respondents’ good will as a finite, expendable resource. Once it’s used up, people will start dropping out of your survey or start satisficing in order to complete it.
Satisficing can be just as bad as when people quit your survey.
When a respondent quits, it’s a bummer. You’re missing out on a possibly crucial opinion that could have sparked your next big idea or contributed to an answer you were dying to know. But here’s the thing: It’s obvious when a respondent drops out. You know exactly who dropped out and when.
Pro Tip: You can use Filters in the Analyze tool to separate out respondents who didn’t complete your survey. This is especially useful when you’ve got a lot of responses and you want to refine your results.
That’s the problem with satisficers who complete your survey: It can be really hard to tell when people are doing it. The responses might appear pretty normal, when in reality, respondents aren’t thinking them through.
They might be putting the bare minimum amount of effort into it, or they just might be just picking option C over and over again because they like the letter (that’s called straightlining and we’ll get to it later).
Before you get paranoid, it’s probably not as common as you think. In this article we’re going to go over the concept of satisficing and a few of the red flags in your survey that will help you spot it.
If want to get down to business, download the free guide to learn how to defeat satisficing in your survey.
Why do people turn to survey satisficing?
We’re all guilty of satisficing, just maybe not in surveys. Think of the last time you went to a restaurant. Did you read every single menu item, balance which one you wanted most, and then select the very best one?
What if you were hungry? How likely would it be that you’d pick the first thing you saw that looked tasty? What if your expectations for the food were pretty low, no matter what you got? What if you were distracted? You might take less time to make your choice.
These same factors apply to survey takers. It really comes down to three issues.
1. Where’s the payoff?
When people invest their time in something, they usually like to have some kind of return on that investment.
While you may get the answers you want from your survey, you might never publish the results and rarely will you share the results directly with respondents. They might see value in contributing their opinion, especially if they’re doing it for the greater good.
That’s why relevancy is so important. People who really care about the subject of the survey have a vested interest in completing it.
2. People are distracted.
This is related, but not quite the same. As a relatively optimistic bunch, we tend to believe that people aren’t motivated purely by self interest. You may be willing to provide well-thought-out feedback, but everyone gets busy, right?
While people may begin your survey with the best of intentions, sometimes they may get interrupted by an important email, or a loud sneeze, or anything else you can imagine. The point is, distracting things happen, and it can really affect how hard people try on surveys.
3. Taking surveys is hard (really!).
Let’s get nerdy for a minute with some survey science mixed with behavioral psychology. Jon Krosnick is a professor of political science, communication, and psychology. He’s written a ton about survey methodology, including about the psychology of why people don’t try hard enough on surveys.
What happens in your respondents’ brain when they’re answering a survey question? According to Krosnick, it should look more or less like this:
- They read the question and process what it’s asking of them
- They search their memory for information that’s relevant
- They summarize all the knowledge they have into one final judgment
- They scan the available answer options to determine which most closely matches their own response, then report that result
When you break it down like that, answering a survey question sounds like kind of a lot of work, right?
Some people cut down on the cognitive strain of answering questions by just not trying very hard on steps 2 and 3. Those people, along with people who are biased during this process, are called weak satisficers.
Strong satisficers, on the other hand, skip steps 2 and 3 altogether. Strong satisficers look for cues in the question that point to the easiest justifiable answer they can give without thinking too much.
Whether it’s strong or weak, satisficing will create bias in your survey data. But if you’re familiar with the types of biases that relate to satisficing, you can easily avoid it.
5 types of bias satisficing introduces into your survey data
Primacy bias: Primacy bias is usually a problem in written or online surveys, so it’s especially relevant to SurveyMonkey. The theory is that people tend to select the first satisfactory answer they find, which is not necessarily the best answer. When too many people are selecting answer option A, you’re probably experiencing primacy bias.
Luckily there’s an easy fix. To avoid primacy bias, simply randomize the answer options in your survey. That way, each respondent sees a different order of answer options.
Recency bias: Recency bias is sort of the opposite of primacy bias, and it’s more of an issue in surveys that are conducted orally rather than visually (think phone surveys vs. online surveys). Recency bias occurs when a respondent chooses the last option they heard or can remember. When people take an online survey all the answer options are in front of them, so recency bias is a lot less of a problem.
So if you suspect you’re running into recency bias with your surveys—ahem—may we suggest online surveys?
Acquiescence bias: Acquiescence bias is a problem because people tend to try to be agreeable, even in surveys. It’s often easier, cognitively speaking, to agree with a statement than it is to disagree with a statement.
Krosnick says that there a ton of personal characteristics that can increase the likelihood of acquiescence bias in surveys—including income, education, knowledge of the topic, and more.
More recently, Researchers at Stanford University in the U.S. and GESIS—Leibniz Institute for the Social Sciences in Germany—used eye-tracking to see how much attention people paid to survey questions. People paid a lot less attention to questions that offered “agree or disagree” as answer options. That means more acquiescence bias.
While we can’t really control for the characteristics Krosnick says can lead to acquiescence bias, we can certainly fix our survey to help control it. To avoid acquiescence bias in surveys, we simply need to avoid questions that use yes/no, agree/disagree, or true/false answer scales and make our surveys easy to take.
“No opinion” bias: OK to be honest, this isn’t an official term. But it represents a real satisficing problem!
Adding a “no opinion” answer option invites satisficing because it gives respondents an easy answer option that they don’t have to think about. It’s best to leave out “no opinion” answer options and let respondents just skip the question if they really have zero opinions about the topic.
Straightlining: Straightlining in surveys is when people choose the same answer option over and over again. It comes from two places—when respondents are being jerks and when respondents are taking a really tough or badly designed survey. Sometimes it’s a little of both.
Straightlining usually happens when there are a lot of very similar questions bunched together and the respondent can’t (or won’t) differentiate between them. Basically, when they see a lot of similar questions, they give a lot of similar responses. This is most commonly a problem in long matrix questions, but it can also happen in surveys with several separate rating questions in a row.
To avoid straightlining in your survey, simply avoid matrix questions and break up long lists of rating questions.
Sometimes you need to survey the right people
While satisficing is a real issue in survey creation, if you follow a few of the best practices outlined above and in our eGuide, you can avoid a lot of it. It’s important to make your survey relatively short, easy to take, and make sure the questions are unambiguous and easy to read.
You can work hard to make your surveys engaging and keep your respondents interested, but sometimes that’s out of your hands. In the end, a lot depends on having respondents who are interested and willing to offer you the time it takes to complete your survey.
SurveyMonkey Audience has millions of people like just like that.
The team regularly assesses the quality of its sample using trusted survey methodology and regular self-profiling surveys keep demographic information up to date. What’s more, U.S. respondents donate $.50 to charity for every survey they take, encouraging thought-through, honest responses.
Want to learn more about running a project on SurveyMonkey Audience? Learn more →
Editor’s note: This blog was originally posted in 2010 and has been revamped for accuracy.
