Survey priming and timing: small changes can have a big impact on bias

It’s possible you’ve heard this one before: Go hungry to a grocery store and you’re likely to come out with a shopping cart full of everything in the store except the refrigeration cases. Go on a full stomach, and you’re more apt to stick to your grocery list.

Why are we talking about shopping? Well, just as a person’s physical state can influence how they shop, a person’s mental state can influence the way they respond to your survey. Their answers—and your results—might differ dramatically based on priming (psychological effects of question order) and timing (when you send your survey).

But with some planning, you can minimize the effects of priming and timing so you get results that accurately reflect your respondents’ attitudes and behaviors. Here’s how.

If you’ve got a decision to make, you know that collecting accurate survey responses will help you make the most informed choice. And if you want to collect honest data, you need to really think about the effects of when you send your survey.

For example, say you work at an electronics manufacturer and your boss asks you to conduct a survey to measure your company’s Net Promoter® Score (NPS), a standard measurement that companies around the world use for benchmarking customer satisfaction and loyalty. The timing of when you send out your survey could yield different results for the question, “How likely is it that you would recommend this company to a friend or colleague?”

If you send out the survey out to customers immediately after they purchase your product, your customer may not have had enough experience with the item to effectively answer whether they’d recommend your company to a friend or colleague. They also might still be in “shiny new toy” mode and give you a higher rating than they would if you sent out the survey a few weeks after their purchase. But, if you wait too long, they may have forgotten all about their experience with the product also making it hard for them to effectively answer that question.

So when should you send your survey? The ideal time to send out your survey is determined by your survey’s goals. Want to assess how their experience was with the purchase process? Send it out immediately after they leave the store. Want to assess their satisfaction with the product? Send your survey a few weeks after their purchase so they’ve had some time to get their hands dirty with the product.

Another issue with timing is that often things happen that you couldn’t plan for in advance, which may impact your results. For example, if there is a recall for a product your company manufactures, the NPS of your company could change dramatically.

In this situation, you can take steps in the analysis process to still have meaningful data. Examine the responses before the recall announcement to get a sense of sentiment before the recall. Take a look at the responses after the recall announcement to determine if the recall had an impact on your customer loyalty. If it did have a negative impact, you still have your previous NPS benchmark to now use as a goal for your company to aspire to.

So that takes care of timing. But what is priming?

Priming definition: In terms of psychology, priming occurs when a person is exposed to a stimulus, and it influences how they respond to another stimulus. For example, if you were to see the word “yellow,” you’d be slightly faster than other people (who hadn’t seen that word) to recognize the word “banana.” That’s because people closely associate words like “yellow” and “banana” in their memory.

Let’s go back to the grocery store example—we mentioned that the state at which you enter into the store might have an effect on your purchases, but the layout of the grocery store may also influence what you buy. If the store displays fresh figs right in the entrance, you’ll be more likely to purchase figs on your trip than you would be if they were only in the produce section.

Similarly, you may be more likely to buy those chips that are right near you in the checkout line than if they were just in the aisle. Surveys are the same–depending on how you lay out your survey or order your questions, your results could differ.

If you specifically mention the device that the individual purchased in the email that invites them to the survey or in the survey introduction, they’ll be “primed” to be thinking about their experience with that specific item. With no mention of the item, they may be thinking about their experience with all the electronics they’ve purchased from your company, which may yield different results.

Think back to your survey goal. Are you trying to get a read on the overall sentiment about your company? Then don’t mention the item purchased. Trying to get a read on how the individual product impacted their feelings about your company? Go ahead and mention it!

We’ve already discussed how images can influence the results, but the order of the questions can also make a difference:

Ask the NPS question first and you get a top-of-the-mind reaction to your company. This helps you get a measure of what someone might say to a friend about you without carefully considering the many different factors that contribute to their overall feeling towards your company.

But what if you ask the NPS question after you’ve asked more specific questions about the service at the store, the check-out experience, the materials the product was made of, and how useful the product is? Well, that means that the respondent is primed to think about all of those factors when responding to the NPS question, giving a more holistic opinion of your company, which may be different from their top-of-the mind view.

Just like the timing issue, the right way to order your survey will depend on your goals and whether you want to get “snap judgments” about your company or whether you’d prefer to get an “informed review.” There’s a case for both.