5 ways to avoid nonresponse errors

What is the difference between nonresponse bias and response bias?

All biases can be placed under two categories: response bias and nonresponse bias. To understand bias it is important to go over both of these categories and explain the differences between them.

Response bias can be defined as the difference between the true values of variables in a study’s net sample group and the values of variables obtained in the results of the same study. This means that response bias is caused by any element in the research that makes its results different from the actual opinions or facts held by the respondents participating in the sample. Most often, this type of bias is caused by respondents giving inaccurate responses and answers being incorrectly recorded or misanalysed.

Nonresponse bias occurs when some respondents included in the sample do not respond. The key difference here is that the error comes from an absence of respondents instead of the collection of erroneous data. Put in more technical terms, nonresponse bias is the variation between the true mean values of the original sample list (people who are sent survey invites) and the true mean values of the net sample (actual respondents). Most often, this form of bias is created by refusals to participate or the inability to reach some respondents.

As discussed in the previous blog on bias and error, to be considered a form of bias a source of error must be systematic in nature. Nonresponse bias is not an exception to this rule. If a survey method or design is created in a way that makes it more likely for certain groups of potential respondents to refuse to participate or be absent during a surveying period, it has created a systematic bias. Take these two examples for instance:

1. Asking for sensitive information: Consider a survey measuring tax payment compliance. Citizens who do not properly follow tax laws will be the most uncomfortable filling out this survey and be more likely to refuse. This will obviously bias the data towards a more law abiding net sample than the original sample. Nonresponse bias in surveys asking for legally sensitive information have been proven to be even more profound if the survey explicitly states that the government or another organization of authority is collecting the data!

2. Invitation issues: Many researchers create nonresponse bias because they do not pretest their invites properly. For example, a large portion of young adults and business sector workers answer the majority of their emails through their smartphones. If the survey invite is provided through an email that doesn’t render well on mobile devices, response rates in smartphone users will drop dramatically. This will create a net sample that under represents the opinions of the smartphone user demographic.

Nonresponse bias is almost impossible to eliminate completely, but there are a few ways to ensure that it is avoided as much as possible. Of course having a professional, well-structured and designed survey will help get higher completion rates, but here is a list of five ways to tweak your research process to ensure that your survey has a low nonresponse bias:

1. Thoroughly pretest your survey mediums: As discussed in the example above, it is very important to ensure that your survey and its invites run smoothly through any medium or on any device your potential respondents might use. People are much more likely to ignore survey requests if loading times are long, questions do not fit properly on their screens, or they have to work to make the survey compatible with their device. The best advice is to acknowledge your sample`s different forms of communication software and devices and pre-test your surveys and invites on each, ensuring your survey runs smoothly for all your respondents.

2. Avoid rushed or short data collection periods: One of the worst things a researcher can do is limit their data collection time in order to comply with a strict deadline. Your study’s level of nonresponse bias will climb dramatically if you are not flexible with the time frames respondents have to answer your survey. Fortunately, flexibility is one of the main advantages to online surveys since they do not require interviews (phone or in person) that must be completed at certain times of the day. However, keeping your survey live for only a few days can still severely limit a potential respondent’s ability to answer. Instead, it is recommended to extend a survey collection period to at least two weeks so that participants can choose any day of the week to respond according to their own busy schedule.

3. Send reminders to potential respondents: Sending a few reminder emails throughout your data collection period has been shown to effectively gather more completed responses. It is best to send your first reminder email midway through the collection period and the second near the end of the collection period. Make sure you do not harass the people on your email list who have already completed your survey!

4. Ensure confidentiality: Any survey that requires information that is personal in nature should include reassurance to respondents that the data collected will be kept completely confidential. This is especially the case in surveys that are focused on sensitive issues. Make certain someone reading your invite understands that the information they provide will be viewed as part the whole sample and not individually scrutinized.

5. Use incentives: Many people refuse to respond to surveys because they feel they do not have the time to spend answering questions. An incentive is usually necessary to motivate people into taking part in your study. Depending on the length of the survey, the difficulty in finding the correct respondents (ie: one-legged, 15th-century spoon collectors), and the information being asked, the incentive can range from minimal to substantial in value. Remember, most respondents won’t have an invested interest in your study and must feel that the survey is worth their time!

Ready for feedback? Send your next survey today. →