Pilot surveys: How to test your survey design in advance

Before you spend time and money running a survey, it pays to do a “dry run” with a small number of respondents. This pilot survey will help you evaluate the strengths and weaknesses of your survey design—including your survey instructions, survey layout, and questions—so you can be confident that you’ll receive the responses you need.

A pilot survey is a test questionnaire that lets you see how people will respond to your survey design and whether your questions will generate useful results.

Pilot surveys are inexpensive, easy to run, and could ultimately save you the trouble of a costly survey rerun in the future. The typical sample size for a pilot survey is just a few dozen respondents, although you can use even fewer respondents if your target audience is small. These respondents also don’t necessarily need to be ideal candidates from your target audience; depending on the type of pilot survey you’re conducting, the way respondents interpret the survey may be more important than the answers they give.

An external pilot survey puts your pilot questionnaire in front of a small group of respondents who won’t be included in your main study. For this type of pilot survey, you can use convenience sampling to gather feedback on your survey design from coworkers, friends, and other people who are easy to recruit. An external study is a great way to test the technical aspects of your design, like your survey instructions and layout.

An internal pilot survey is a small preliminary study that precedes your main study. Unless the internal pilot survey reveals something really wrong with your survey, the results are ultimately included in your main study and the respondents are considered your first participants. Since the respondents are always pulled from your target audience, an internal pilot survey is more realistic in that it helps you understand the type of responses you’re likely to get once you send your survey to the larger group.

Respondents who take part in a participatory pilot survey know they’re part of a pilot study. Because respondents understand this isn’t a “real” study, you can ask them for feedback on your survey design with questions specific to the pilot questionnaire. For example, you might ask participants to rate the clarity of your instructions or the difficulty of the questions.

On the other hand, respondents who participate in an undeclared pilot survey don’t realize they’re taking part in a pilot study. These respondents believe the pilot questionnaire is the real survey, so they focus more on the content of your questions than the design of your survey. An undeclared pilot survey generates results that can help you predict the outcome of your main survey.

In their seminal paper on survey questions, Converse and Presser (1986) recommend an approach to pilot surveys called “respondent debriefing.” This approach uses the undeclared pilot survey method by presenting a pilot questionnaire as the main survey. After respondents complete the questionnaire, you reveal that they were part of a pilot study and follow up with questions you’d typically include in a participatory pilot survey.

Keep in mind that acceptable methods for running a pilot survey in research vary based on the subject of your research. If standard practices for pilot studies exist in your field of study or industry, be sure to take them into account in your pilot survey.

When your pilot study is complete, you’re ready to use the results to build a better survey. Answers from participatory questions can help you identify and remedy any design and content issues that really stood out to respondents. The results of your actual survey questions also give you useful insights into the way people interpret your questions. Go through your survey and make updates based on what you learned from your pilot survey. If needed, you can always conduct a second pilot survey to see whether the issues were resolved. 

Even if you don’t conduct a full pilot survey, it’s important to test your survey design before you send your survey to respondents. Learn more about previewing and testing your survey with SurveyMonkey.