When our research team at SurveyMonkey releases a poll with one of our media partners or publishes our weekly Trump approval numbers, we’re using data collected with our unique methodology. About 3 million people take user-generated surveys on SurveyMonkey every day, and immediately after they do so we invite a small percentage of them to take one of our own surveys.
The number of respondents we get to our surveys, and the pace at which they roll in, varies significantly by time of day and day of week—as does traffic to all SurveyMonkey surveys and all web traffic generally. For example, people tend to sleep during night hours; we’ve always known that we get fewer responses overnight than we do over the course of a 9am – 5pm work day.
But, one big question is whether this natural time of day variation has any effect on the responses we get. Traditionally, polling firms have relied on surveys conducted on the telephone, which limits the hours of the day during which they can reach respondents. It’s rare to get a phone call from a pollster before 9am (though it’s notoriously common to get one right as you’re sitting down to dinner with your family). By expanding the number of hours per day that we field surveys, are we reaching a different population of respondents? Are the people who respond at 9am somehow different from the people who respond at 5pm—or 2am?
To answer these questions, we took all the data we collected from our SurveyMonkey Election Tracking polls during the five weeks immediately prior to the 2016 U.S. election. That’s data from a total of 219,431 respondents! For all respondents, we examined basic demographic information (age, sex, race, education, state, etc.), partisanship (whether they identify as a Republican or Democrat or independent), and vote intention (whether they planned to vote for Hillary Clinton or Donald Trump). We also looked at relevant paradata, such as the exact time they completed our survey and the device they used to complete it (an iPhone, a desktop PC, etc.).
Armed with this data, we could examine a full range of responses over the course of a day, a week, or more. We discovered some interesting patterns, and after presenting this research at the most recent meeting of the American Association for Public Opinion Research (AAPOR), we wanted to share our findings with our blog readers, too.
Weekend days look different
Each of the graphs that follow show what percentage of responses occur at different times of day, from midnight (12am) to 11:59pm. The first graph makes one thing pretty clear: all weekdays look very similar, while weekend days have their own unique patterns.
Of all the people who responded to one of our surveys on a Monday in the five weeks prior to the election, less than 2 percent did so during each one hour block of time from midnight to 6am. On Mondays, responses start to pick up around 7am, then continue throughout the morning until reaching a peak at 10am, dropping off during the lunch hours, peaking even higher at 2pm, then dropping off again until the next morning. Data for Tuesdays, Wednesdays, Thursdays, and Fridays all follow this familiar pattern, with responses increasing throughout the morning hours, declining at lunchtime, rising again mid-afternoon and then falling through the evening hours.
Weekend days, however, look quite different! Responses on Saturdays rise through the early morning and reach a peak around 10am, just as on weekdays, but then they trail off through the rest of the day and never recover. People just seem to have better things to do on Saturday afternoons and evenings rather than answering surveys. On the other hand, responses on Sundays climb slowly but steadily throughout the course of the day, not peaking until 9pm—possibly as people are logging online to prepare for the week ahead.
Demographics
Pollsters typically analyze demographic characteristics—like age, sex, race, and education—to make sure they’re getting a representative sample of respondents. We do the same with our SurveyMonkey polls, but we still wanted to see whether we could discern any response pattern.
As the charts above show, there are only very minor differences in time of response by different demographic characteristics. The proportion of men who respond to surveys at any particular time on a weekday is nearly identical to the proportion of women who are responding at that time. Younger respondents ages 18-34 get a slightly later start to their day (in terms of survey-taking), but they still follow the same pattern as older respondents. Similarly, respondents of different races and different educational backgrounds show almost no variation in their response patterns through the course of a typical weekday.
Politics
As with demographics, we see very little variation in response by time of day in terms of partisanship and politics. If you thought that Republicans get up early to watch Fox and Friends or Democrats stay up late to watch Rachel Maddow, our data can provide no evidence for those claims. We see a similar distribution of respondents by party identification during each hour of the day, and we see the same pattern among respondents who said they planned to vote for Clinton as those who planned to vote for Trump.
Big differences in mobile responses
Although there are only minor differences by demographics and politics, there’s one factor that does show a large variation in response time across the course of a typical day: device type. As more and more people take surveys on their mobile devices—35% of SurveyMonkey respondents in the U.S., at our last count—it’s clear that mobile respondents have very different response patterns compared to traditional desktop or laptop respondents.
On weekend days, there is very little difference in response patterns by people answering surveys on mobile devices compared to those answering on non-mobile devices. But, the story is very different on weekdays! During the week, most of our SurveyMonkey respondents are completing surveys on desktop or laptop computers, and therefore the distribution of all non-mobile respondents on a typical weekday looks very similar to the typical weekday distribution discussed above, with peaks at 10am and 2pm. But, the distribution of mobile respondents during the week looks very different, with the trend line continuously rising throughout the course of the day, plateauing in the afternoon hours, then slowly dropping off in the evenings. All in all, this means that respondents during the day are more likely to be answering on a computer, while respondents in the late afternoon and evening are more likely to be answering on a mobile device.
Why is this important?
People tend to have predictable patterns to their days: get up, work, eat, work some more, spend time with family or friends, go to bed, do the same thing all over again. As it turns out, we see similar patterns when it comes to taking surveys, too. If we can better understand these patterns—in particular those related to mobile responses—maybe we can do a better job of reaching people when it’s optimal for them.
