We’re still a few days away from the weekend and we’ve already got cake on the brain, dear survey fans.
Comparisons are a basic impulse. Tell your friend you like angel food cake and she might counter, “Really?! More than birthday cake? Or carrot cake?” Naturally, this will turn into a heated discussion over how to rank the deliciousness of these kinds of cakes.
Your survey might look a little something like this practice survey. We whipped it up just for you, so feel free to get in there and start ranking!
The problem: A ranking question may seem like the natural fit, however, there is a drawback to using a ranking question. Although ranking different kinds of cake gives you a relative sense of whether one is liked more than the other—you don’t know how much more. Someone may like chocolate cake only slightly more than carrot cake, but thinks cheesecake is disgusting. A ranking question will not let you know the strength of preferences—only the order.
Let’s stick to the sweets to help illustrate another shortcoming of ranking questions, shall we?
M&Ms come in six colors, so your ranking question might look like this.
Now, almost everyone has a favorite and a least favorite color M&M, but it’s somewhere in the middle where it starts to get a little blurry. Red is towards the top and brown towards the bottom, but as for whether yellow is better than green or vice versa? It’s all kind of a toss-up. Could depend on your mood, the color of your outfit, or maybe it just doesn’t matter at all.
The moral of the sweet story here? Ranking questions can get difficult in a hurry. Once you get past three choices in a ranking question, it gets downright sloppy and confusing—and psychological research has proven that it’s a very tough thing for people to rank more than three options.
The problem: When you give people a cognitively difficult task like ranking question choices of say, six things, often they will end up getting frustrated and will start to rank choices randomly. This will lead to higher dropout rates from your surveys and even worse, messy data for you to analyze from those who do manage to power through and finish. Remember, although ranking questions gives you a relative sense of whether one thing is liked more than the other—you don’t know how much more.
So, what can you do instead? Well, that depends on why you’re asking! Just one more reason you need to decide why you’re asking before you come up with how to ask it.
Use Case #1:
- Sample Research Question: The Mars company wants to double the number of the favorite color of M&M and eliminate the least favorite color of M&M. To know which to double and which to keep, Mars needs to know which is the best and which is the worst.
- Sample Survey Solution: If you just want to know what people’s favorite and least favorite thing is and you don’t really care about all that middle stuff, try splitting it into two closed-ended questions:
The color that is chosen the most times in the favorite question is the winner you’ve been looking for, and the color that is chosen the most times in the least favorite question is the loser you’ll need to eliminate.
Use Case #2:
- Sample Research Question: The Mars company wants to add MSG to the less-liked colors of M&Ms to give them a flavor boost so that consumers find the colors all equally tasty. To know how much MSG to add, Mars needs to know the relative standing of the colors.
- Sample Survey Solution: If you really do care about the relative standing of all six colors of M&Ms (or anything else non-candy related), split up the one ranking question into six rating questions:
Respondents rate each of the six M&M colors one at a time. Using these separate ratings, you can then compute an average score for each M&M. You then have an overall ranking of liking for each color based on the average score for each color.
So what’s the best approach to use overall? Every question type has its strengths and weaknesses. It’s just a matter of knowing your research question. If you know why you’re asking the question, the best trade-off for your particular survey should be pretty clear. So with that, it’s off to chow down on some cake and M&Ms!
