Products

SurveyMonkey is built to handle every use case and need. Explore our product to learn how SurveyMonkey can work for you.

Get data-driven insights from a global leader in online surveys.

Explore core features and advanced tools in one powerful platform.

Build and customise online forms to collect info and payments.

Integrate with 100+ apps and plug-ins to get more done.

Purpose-built solutions for all of your market research needs.

Create better surveys and spot insights quickly with built-in AI.

Templates

Measure customer satisfaction and loyalty for your business.

Learn what makes customers happy and turn them into advocates.

Get actionable insights to improve the user experience.

Collect contact information from prospects, invitees, and more.

Easily collect and track RSVPs for your next event.

Find out what attendees want so that you can improve your next event.

Uncover insights to boost engagement and drive better results.

Get feedback from your attendees so you can run better meetings.

Use peer feedback to help improve employee performance.

Create better courses and improve teaching methods.

Learn how students rate the course material and its presentation.

Find out what your customers think about your new product ideas.

Resources

Best practices for using surveys and survey data

Our blog about surveys, tips for business, and more.

Tutorials and how to guides for using SurveyMonkey.

How top brands drive growth with SurveyMonkey.

Contact SalesLog in
Contact SalesLog in
Survey Science

Let’s agree NOT to use agree/disagree questions

Let’s agree NOT to use agree/disagree questions

Even in tough situations, people generally try to be positive and agreeable.

While that’s often a virtue in life, it isn’t when it comes to surveys. In fact, it can actually introduce bias in your survey.

Researchers—like my team here at SurveyMonkey—have demonstrated that there are several ways that agree/disagree questions can cause your respondents to answer in a way that doesn’t always reflect their true opinions. Let’s take a look.

Here are two very similar questions that ask pretty much the same thing.

You might think that by asking either question, you’d get identical results, right? Respectfully, we disagree. There are two major types of problems that can arise from using agree/disagree questions.

Acquiescence bias: People have the tendency to say they like things, to say “yes” to things, or to agree with things—even if they don’t actually feel that way. Think about it this way: When your coworker asks if you like her new sweater, you’ll usually agree with her, right? Research has shown that tendency to hold true in answering survey questions, too.

Straightlining: Another way that agree/disagree questions cause problems is by making it easy to go through many questions of a survey and select the same answer every time without actually reading the options. Having multiple questions in a row with identical answer options (like agree/disagree) can bore your respondents. They may still be responding to each question, but they’re not carefully considering which response is best.

Here’s new evidence that shows how little attention respondents are paying to your agree/disagree answer scale. Researchers at Stanford University in the U.S. and GESIS – Leibniz Institute for the Social Sciences in Germany used eye-tracking software to detect where respondents were looking as they answered survey questions online.

They did an experiment asking similar questions in two different formats—agree/disagree and item-specific—and they presented their results at the annual meeting of the Pacific Chapter of the American Association for Public Opinion Research.

Item-specific questions have response options that are specific to a particular question. Instead of having a one-size-fits-all scale that applies to many different agree/disagree questions, they use a scale whose wording is tailored to match the unique question wording.

Here’s an example of a pair of questions the researchers used in their experiment. Notice how both questions ask for the same information—whether someone is interested in politics—using different question wordings and different answer formats, each with five choices.

The researchers used eye-tracking technology to track respondents’ attention to each type of question. The results show that survey-takers read and comprehend both types of questions in the same way, but they process the answer choices very differently.

The researchers measured:

  1. the number of fixations, or instances where someone is looking at a specific point;
  2. the duration of each fixation; and
  3. the number of re-fixations, or instances where someone switched his gaze from one answer option to a different one.

They found that respondents to the item-specific version of the question had higher fixation counts, longer fixation lengths, and more re-fixation instances than respondents to the agree-disagree version of the same question.

In plain English, that means that people answering questions with item-specific response options took more time considering each possible response, looking back and forth between options before deciding which best matched their own opinions. This is exactly what respondents should be doing as they take a survey.

If your survey bores your respondents by asking them a series of agree/disagree questions, this research demonstrates that they aren’t going to be particularly focused or conscientious when they’re responding. And what’s the point of asking a question if you know your respondents aren’t really going to try to answer it well?

Item-specific questions require a little more effort to answer than agree/disagree questions do. But don’t worry—the benefits outweigh the negatives. More effort means more concentration and more care while responding, which in turn leads to more well-considered responses and more accurate data.

SurveyMonkey works hard to ensure everything about the survey-making and survey-taking experience is easy; you can trust us when we say that in this case, it’s better to make things just a tiny bit more difficult.

This information comes from a study by Jan Karem Höhne (Stanford University, USA) and Timo Lenzner (GESISLeibniz Institute for the Social Sciences in Germany).