Exploring the methodology behind SurveyMonkey’s concept testing solutions

The results of a concept testing study can be incredibly powerful. 

If done correctly, they can ensure you take the absolute best idea to market, whether its an ad, logo, product idea, video—really anything. But if your methodology is sloppy, false positives (or negatives) can arise in your data that could result in potentially costly bad decisions.

When it comes to your data, we’re not leaving anything up to chance. That’s why we’ve built our concept testing and creative analysis solutions around monadic survey design: We believe it to be the most trustworthy and methodologically sound way to set up a concept testing survey.

For context, there are three common ways to set up a concept testing survey.

Why do we think monadic survey design is better? It comes down to four main reasons.

1. It’s similar to the real world: In the end, your customers will see only one final version of your idea—not several. In a monadic survey design, you’re effectively setting up a testing environment that mirrors what your customers will experience when your idea is out in the market. Since they show multiple versions of the same concept, forced choice and sequential monadic survey designs don’t offer this benefit.

2. It results in better data: Unless your survey is as engaging as a trending YouTube video, you should never expect respondents to spend more than 10 minutes to answer a 30-question survey. 

Even when respondents are fairly compensated, the quality of their responses tend to drop dramatically in long surveys. Respondents start to lose patience and stop providing thoughtful responses.

When it comes to concept testing surveys, survey fatigue usually get worse. In addition to reading your question text and answer options, respondents must pay close attention to your logo, read your product description carefully, or even watch a full video. It’s important to factor in this extra burden on respondents when you’re deciding on how you should design a concept testing survey.

Using monadic design is the most effective way to limit respondent burden in a concept testing survey. Respondents only need to evaluate one stimulus, which gives them space to be more thoughtful and give your better data. 

Furthermore, because they only need to evaluate one stimulus, you can also dive deeper and ask them more follow-up questions than you could in other designs. In most cases, it’s not an issue to ask as many as 10 follow-up questions that touch on various aspects of a stimulus like visual design, information, sentiment, action, and more. 

In this way, sequential monadic designs are the most problematic. Respondents will have to answer the same set of questions over and over again for each stimulus that they are exposed to. If you have 3 stimuli for each respondent, and 10 questions for each stimulus, the total number of questions multiplies quickly to 30, which is sure to affect response quality.

3. It eliminates the potential for order bias: Order bias can affect the results of any type of survey, including concept testing surveys. The two most common types, primacy effect and anchoring effect, are particularly important when it comes to concept testing. 

Studies have shown that people evaluate the first stimulus they see in a concept testing survey more favorably than subsequent stimuli. In other words, if you’re using sequential monadic design, the primacy effect will give the first stimulus respondents see an extra advantage.

The anchoring effect can have a similar impact on your results. Studies show that the quality of a previous stimulus can have an effect on how people rate the subsequent stimuli they see, which means if the first stimulus happens to be a good one, the subsequent stimuli will likely be evaluated less positively.

The more stimuli that you have, the more likely you’ll encounter an uneven split. To get rid of order bias completely, monadic design is the way to go.

Some might argue that you can randomize the order of stimuli to control for these problems. However, unless you have a large sample size, this method isn’t guaranteed to work. Just like you might get 3 heads or 7 heads when you flip a coin 10 times, randomly showing one of the two stimuli first to a small number of respondents might not get an even split. Whichever stimulus gets shown to your respondents first more often will instantly get an advantage. 

4. It allows for ad hoc benchmarking studies:  Let’s say your competitor has a very successful ad campaign that you’d like to beat and you want to find out if the ads you come up with to beat it will actually perform better. One solution is to run a concept testing study that tests your ad ideas against your competitors’ ads.

Here’s the thing. If you use a forced choice or sequential monadic design, your competitor’s campaign might get an advantage simply because the ads in the campaign will likely be very different from your ads. 

Imagine if respondents saw two of your ads and one of your competitors’ in the same survey. The ad that stands out (your competitor’s) will gete a natural advantage because it will draw much more attention. To ensure fair benchmarking, you’ll need to choose the monadic design where every ad will be evaluated individually and not in the context of other ad.

No matter what you use concept testing for, the results of your study are bound to be critical. The scariest thing about running a concept testing study is that you may not know when things have gone awry, which means you could be making major business decisions based on flawed data. That’s why it’s best to make sure you’re addressing every possible issue that might come up in your study. 

The survey methodology we’ve adopted for our concept testing solutions is designed to give you the best data possible. You should have the confidence that your next ad campaign will blow your competitor’s out of the water and be sure your product idea will be a winner—we wouldn’t accept anything less.