How to analyse survey data

The results are back from your online surveys. Now it’s time to harness the power of survey data analysis to make sense of the results and present them in ways that are easy to understand and act on. After you’ve collected statistical survey results and have a data analysis plan, it’s time to start the process of calculating the survey results you received back. Here’s how our survey research scientists make sense of quantitative data (versus qualitative data). They structure their reporting around survey responses that will answer research questions. Even for the experts, it can be hard to parse the insights in raw data. 

In order to reach your survey goals, you’ll want to start off by relying on the survey methodology that our experts suggest. Then, once you have results, you can effectively analyse them using all the data analysis tools available to you, including statistical analysis, data analytics, and charts and graphs that capture your survey metrics.

Sound survey data analysis is key to getting the information and insights you need to make better business decisions. Yet it’s important to be aware of potential challenges that can make analysis more difficult or even skew results. 

Asking too many open-ended questions can add time and complexity to your analysis because it produces qualitative results that aren’t numerically based. Meanwhile, closed-ended questions generate results that are easier to analyse. Analysis can also be hampered by asking leading or biased questions or posing questions that are confusing or too complex. Being equipped with the right tools and know-how helps ensure that survey analysis is both easy and effective.

Read more about using closed-ended vs. open-ended questions.

With its many data analysis techniques, SurveyMonkey makes it easy for you to turn your raw data into actionable insights presented in easy-to-grasp formats. Features such as automatic charts and graphs and word clouds help bring data to life. For instance, Sentiment Analysis allows you to get an instant summary of how people feel from thousands or even millions of open text responses. You can review positive, neutral and negative sentiments at a glance or filter by sentiment to identify areas that need attention. For even deeper insights, you can filter a question by sentiment. Imagine being able to turn all those text responses into a quantitative data set.

Word clouds let you quickly interpret open-ended responses through a visual display of the most frequently used words. You can customise the look of your word clouds in a range of ways, from selecting colours or fonts for specific words to easily hiding non-relevant words.

Our wide range of features and tools can help you address analysis challenges and quickly generate graphics and robust reports. Check out how a last-minute report request can be met in a jiffy using SurveyMonkey.

First of all, let’s talk about how you’d go about calculating survey results from your top research questions. Did you feature empirical research questions? Did you consider probability sampling? Remember that you should have outlined your top research questions when you set a goal for your survey.

For example, if you held an education conference and gave attendees a post-event feedback survey, one of your top research questions may look like this: How did the attendees rate the conference overall? Now take a look at the answers you collected for a specific survey question that addresses that top research question:

Do you plan to attend this conference next year?

Note that you have some percentages (71%, 18%) and some raw numbers (852, 216) in the responses. The percentages are just that: the percentage of people who gave a particular answer. In other words, the percentages represent the number of people who gave each answer as a proportion of the number of people who answered the question. So, 71% of your survey respondents (852 of the 1,200 surveyed) plan to come back next year.

This table also shows you that 18% say they are planning to return and 11% say they are not sure.

Having a good understanding of sample size is also key to making sure you are accurately and effectively analysing your survey results. Sample size is how many people you need to take your survey and complete responses to make it statistically viable. Even if you’re a statistician, determining survey sample size can be a challenge. But SurveyMonkey takes the guesswork and complexity out of the process with its easy-to-use margin of error calculator that helps you determine how many people you need to survey to ensure that your results help you avoid your margin of error.

You'll remember that when you set a goal for your survey and developed your analysis plan, you thought about which subgroups you were going to analyse and compare. Now is when that planning pays off. You might, for example, want to see how teachers, students and administrators compared to one another in terms of their responses about attending next year’s conference. To work this out, you want to dive into response rates by means of crosstabulation, or use crosstab reports, where you show the results of the conference question according to subgroup:

From this table, you can see that a large majority of the students (86%) and teachers (80%) plan to come back next year. However, the administrators who attended your conference look different, with under half (46%) of them intending to come back. Hopefully, some of our other questions will help you work out why this is the case and what you can do to improve the conference for administrators so that more of them will return year after year.

A filter is another method of data analysis when you’re modelling data. Filtering means narrowing down your focus to one particular subgroup and filtering out the others. So, instead of comparing subgroups to one another, here we’re just looking at how one subgroup answered the question. Combining filters can give you pinpoint accuracy in your data.

For instance, you could limit your focus to just women, or just men, then re-run the crosstab by type of attendee to compare female administrators, female teachers and female students. One thing to be wary of as you slice and dice your results: Every time you apply a filter or crosstab, your sample size decreases. To make sure your results are statistically significant, it may be helpful to use a sample size calculator.

Graphs can be a regular go-to tool when you aim to quickly demonstrate the results of your data analysis in a way that is easy for anyone to understand. With SurveyMonkey, it’s easy to create graphs that provide clarity and context to your analysis which, in turn, makes it possible to use the data in more targeted and actionable ways.  

Crosstabulations (otherwise known as crosstab reports) are useful tools for delving more deeply into your data. Crosstabs structure your data into a table that groups respondents based on shared background information or survey responses, allowing you to compare each group’s answers to one another. This helps you better understand each group of respondents and uncover how they differ from each other.

Let’s say one key question on your conference feedback survey is: “Overall, how satisfied were you with the conference?” 

Your results show that 75% of the attendees were satisfied with the conference. That sounds quite good. But wouldn’t you like to have some context? Something to compare it against? Is that better or worse than last year? How does it compare to other conferences?

Benchmarking can provide answers to these questions and more by readily allowing you to make comparisons to past and current data to identify trends in your industry and marketplace and see how you stack up against them.

Well, if you did ask this question in your conference feedback survey after last year’s conference, you’d be able to make a trend comparison. Although professional pollsters aren't exactly known for their wit, one of their favourite maxims is “trend is your friend”. If last year’s satisfaction rate was 60%, this means you've increased satisfaction by 15 percentage points! What caused this increase in satisfaction? Hopefully, the responses to other questions in your survey will provide some answers.

If you don’t have data from prior years’ conferences, make this the year you start collecting feedback after every conference. This is called benchmarking. You establish a benchmark or baseline number and, in the future, you can see whether and how this has changed. You can benchmark not just attendees’ satisfaction, but other questions as well. You’ll be able to track, year after year, what attendees think of the conference. This is called longitudinal data analysis.

You can even track data for different subgroups. If, for example, satisfaction rates are increasing year on year for students and teachers but not for administrators, you might want to look at administrators’ responses to various questions to see if you can gain an insight into why they are less satisfied than other attendees.

You know how many people said they were coming back, but how do you know if your survey has yielded answers that you can trust and answers that you can use with confidence to inform future decisions? It’s important to pay attention to the quality of your data and to understand the components of statistical significance.

In everyday conversation, the word 'significant' means important or meaningful. In survey analysis and statistics, significant means 'an assessment of accuracy'. This is where the inevitable 'plus or minus' comes into survey work. In particular, it means that survey results are accurate within a certain confidence level and not due to random chance. Drawing an inference based on results that are inaccurate (i.e. not statistically significant) is risky. The first factor to consider in any assessment of statistical significance is the representativeness of your sample, i.e. to what extent the group of people who were included in your survey 'look like' the total population of people about whom you want to draw conclusions.

You have a problem if 90% of conference attendees who completed the survey were men yet only 15% of all your conference attendees were male. The more you know about the population you are interested in studying, the more confident you can be when your survey lines up with those numbers. When it comes to gender, at least, you’ll be quite happy if men make up 15% of survey respondents in this example.

If your survey sample is a random selection from a known population, statistical significance can be calculated in a straightforward manner. A primary factor here is sample size. Suppose 50 of the 1,000 people who attended your conference replied to the survey. Fifty (50) is a small sample size and results in a broad margin of error. In short, your results won’t carry much weight.

Say you asked your survey respondents how many of the 10 available sessions they attended over the course of the conference. And your results look like this:

You might want to analyse the average. As you may recall, there are three different kinds of averages: mean, median and mode.

In the table above, the average number of sessions attended is 6.1. The average reported here is the mean, which is the type of average that’s probably most familiar to you. To determine the mean, you add up the data and divide that by the number of figures you added. In this example, you have 100 people saying they attended one session, 50 people for four sessions and 100 people for five sessions, etc. So, you multiply all of these pairs together, add them up and divide by the total number of people.

The median is another type of average. The median is the middle value: the 50% mark. In the table above, we would locate the number of sessions where 500 people were to the left of the number and 500 were to the right. The median is, in this case, six sessions. This can help you eliminate the influence of outliers, which may adversely affect your data.

The last type of average is mode. The mode is the most frequent response. In this case, the answer is six. 260 survey participants attended six sessions, more than attended any other number of sessions.

Means and other types of averages can also be used if your results were based on Likert scales.

When it comes to reporting on survey results, think about the story the data tells.

Let's suppose that, overall, your conference received mediocre ratings. You dig deeper to find out what’s going on. The data show that attendees gave very high ratings to almost all the aspects of your conference (the sessions and classes, the social events and the hotel) but that they really disliked the city chosen for the conference. (Maybe the conference was held in Aberdeen in January and it was too cold for anyone to go outside!) 

That is part of the story right there: it was a great conference overall, but the location was totally unsuitable. Brighton, Nice or Barcelona might be a better choice for a winter conference.

One aspect of data analysis and reporting you have to consider is causation vs. correlation.

People digest and understand information in a range of different ways. Fortunately, SurveyMonkey offers an abundance of different ways for you to analyse survey data so you can assess and present the information in ways that will be most useful to meet your goals and create graphs, charts and reports that make your results easy to understand.

Here are some of the common questions that we can help you navigate as you build up your survey analysis expertise:

Longitudinal data analysis (often called 'trend analysis') is basically tracking how findings for specific questions change over time. Once a benchmark is established, you can determine whether and how numbers shift. Let's suppose that the satisfaction rate for your conference was 50% three years ago, 55% two years ago, 65% last year and 75% this year. In this case, congratulations are in order because your longitudinal data analysis shows a solid, upward trend in satisfaction.

Go back

Causation is when one factor causes another, while correlation is when two variables move together, but one does not influence or cause the other. For example, drinking hot chocolate and wearing mittens are two variables that are correlated, in that they tend to go up and down together. However, one does not cause the other. In fact, they are both caused by a third factor: cold weather. 

Cold weather influences both hot chocolate consumption and the likelihood of wearing mittens. Cold weather is the independent variable, and hot chocolate consumption and the likelihood of wearing mittens are the dependent variables. In the case of our conference feedback survey, the cold weather probably influenced attendees' dissatisfaction with the conference city and the conference overall. 

Finally, to further examine the relationship between variables in your survey, you may need to perform a regression analysis.

Go back

Regression analysis is an advanced method of data visualisation and analysis that allows you to look at the relationship between two or more variables. There are many types of regression analysis and the one(s) a survey scientist chooses will depend on the variables they are examining. What all types of regression analysis have in common is that they look at the influence of one or more independent variables on a dependent variable. In analysing our survey data, we might be interested in knowing what factors have the greatest impact on attendees’ satisfaction with the conference. Is it a matter of the number of sessions? The keynote speaker? The social events? The site? Using regression analysis, a survey scientist can determine whether and to what extent satisfaction with these different attributes of the conference contribute to overall satisfaction.

This, in turn, provides insight into which aspects of the conference you might want to alter next time around. Let's suppose, for example, that you paid a hefty fee to secure the services of a top-flight keynote speaker for your opening session. Participants gave this speaker and the conference overall high marks. Based on these two facts, you might think that securing the services of a fabulous (and expensive) keynote speaker is the key to conference success. Regression analysis can help you determine whether this is indeed the case. You might find that the popularity of the keynote speaker was a major driver of satisfaction with the conference. If so, next year you’ll want to secure the services of a great keynote speaker again. However, if for example, the regression shows that although everyone liked the speaker, this did not contribute much to attendees’ satisfaction with the conference, the large sum of money spent on the speaker might be better spent elsewhere. 

If you take the time to carefully analyse the soundness of your survey data, you’ll be on your way to using the answers to help you make informed decisions.

Survey data can be one of your most powerful tools

By analysing data in fresh, engaging and insightful ways, you can help drive your company’s growth, deepen customer relationships and stay steps ahead of the competition. SurveyMonkey has a range of options to meet any budget. 

Survey data collection uses surveys to gather information from specific respondents. Survey data collection can replace or supplement other data collection types, including interviews, focus groups and more. The data collected from surveys can be used to boost employee engagement, understand buyer behaviour and improve customer experiences.

Go back

Longitudinal data analysis (often called 'trend analysis') is basically tracking how findings for specific questions change over time. Once a benchmark is established, you can determine whether and how numbers shift. Let's suppose that the satisfaction rate for your conference was 50% three years ago, 55% two years ago, 65% last year and 75% this year. In this case, congratulations are in order because your longitudinal data analysis shows a solid, upward trend in satisfaction.

Go back

Causation is when one factor causes another, whereas correlation is when two variables move together but one does not influence or cause the other. For example, drinking hot chocolate and wearing a woolly hat are two variables that are correlated, in that they tend to go up and down together; however, one does not cause the other. In fact, they are both caused by a third factor: cold weather. Cold weather influences both hot chocolate consumption and the likelihood of wearing a woolly hat. Cold weather is the independent variable, and hot chocolate consumption and the likelihood of wearing a woolly hat are the dependent variables. In the case of our conference feedback survey, cold weather most probably influenced attendees' dissatisfaction with the conference city and the conference overall. Finally, to further examine the relationship between variables in your survey, you might need to perform a regression analysis.

Go back

Regression analysis is an advanced method of data visualisation and analysis that allows you to look at the relationship between two or more variables. There are many types of regression analysis and the one(s) a survey scientist chooses will depend on the variables he or she is examining. What all types of regression analysis have in common is that they look at the influence of one or more independent variables on a dependent variable. In analysing our survey data, we might be interested in knowing what factors have the greatest impact on attendees’ satisfaction with the conference. Is it a matter of the number of sessions? The keynote speaker? The social events? The site? Using regression analysis, a survey scientist can determine whether and to what extent satisfaction with these different attributes of the conference contribute to overall satisfaction.

This, in turn, provides insight into which aspects of the conference you might want to alter next time around. Let's suppose, for example, that you paid a hefty fee to secure the services of a top-flight keynote speaker for your opening session. Participants gave this speaker and the conference overall high marks. Based on these two facts, you might think that securing the services of a fabulous (and expensive) keynote speaker is the key to conference success. Regression analysis can help you determine whether this is indeed the case. You might find that the popularity of the keynote speaker was a major driver of satisfaction with the conference. If so, next year you’ll want to secure the services of a great keynote speaker again. However, if for example, the regression shows that although everyone liked the speaker, this did not contribute much to attendees’ satisfaction with the conference, the large sum of money spent on the speaker might be better spent elsewhere. If you take the time to carefully analyse the soundness of your survey data, you’ll be on your way to using the answers to help you make informed decisions.