Non-probability sampling

Running a large-scale survey can be a tricky task. As much as you need your results to represent the entire population, it’s hard to give everyone you’d like to hear from the chance to be surveyed.

One real-world solution is to use non-probability sampling, which SurveyMonkey knows a thing or two about. With more than half a million people available to take surveys through our Audience panel at any time, SurveyMonkey has the largest non-probability sample in the US.

Non-probability sampling selects a group of respondents from a larger population, knowing full well that some members of the population have zero chance of being surveyed. This is not allowed in probability sampling, which requires everyone in the population to have a non-zero chance of being selected.

Whether you’re using a panel like SurveyMonkey Audience or any other type of non-probability sampling design, the way you select respondents will always knowingly leave out some members of the population.

Sometimes these exclusions are obvious, like when people can opt in or out of responding to you. For example, you might ask your customers to provide their emails so that they can participate in a customer feedback survey. Some will probably decline, which means they have no chance of being selected into your survey sample.

Other times, these exclusions are subtler. Let’s say you plan to survey the first 100 people who walk into your store on a given day. This may seem like a random probability sampling design, but consider this: there’s probably a difference between the type of people who can come to your store in the morning versus those who have to visit later.

If your store opens at 9 am, maybe your first customers of the day are less likely to be employed than those who are coming in at 7 or 8 at night. Since some part of the population has no chance of being among those first 100 customers of the day, your results may be biased—so you’re actually using non-probability sampling here.

Here are some non-probability sampling designs that are used regularly, even if they’re not well-suited to all surveys:

Probability sampling is favored by statisticians, but for people conducting surveys in the real world, non-probability sampling is more practical. If done well, non-probability sampling can give you the same (or better) high-quality data you would expect from a true probability sample.

Most surveys are targeted at a very specific population and don’t need to ensure the same diversity and representation provided by probability sampling. If you’re doing market research on mothers of young children, you don’t need a probability sample that includes men, people without children, or people with adult children.

Even when a non-probability sample doesn’t perfectly overlap with your population of interest, there are still plenty of advantages to sticking with it.

Getting responses with non-probability sampling is usually faster and cheaper than getting them with probability sampling, because sample members are more motivated to respond than people who are randomly contacted. People selected from a mailing list, for example, are probably more loyal to a company than people chosen from outside of it.

The biggest challenge of non-probability sampling is recreating the same kind of non-biased results that probability sampling gives you.

Always be careful that the way you’re recruiting respondents isn’t distorting your data. Some online panels pay their respondents, which can lead to bias from “professional” survey-takers who answer just for the money and are not providing accurate information.

When you’re doing a non-probability survey, be sure to think through potential sources of bias. It’s not always easy to predict what will bias your results, but starting with a diverse group of respondents with characteristics that match your population of interest is essential. This will not only give you data that is just as accurate as probability sampling, but it will be much more cost- and time-effective, too.