Online surveys: Can we trust their results?

During this run-up to the presidential election, results from voter surveys (or “polls”) are everywhere you turn. It’s always good practice to read the fine print on the specific method used, though, since it’s very easy to create a survey that produces suspect data. Fortunately, many of the major news organizations have gotten markedly better about disclosing details about samples, response options, margins of error, and so on.

Surveys are vital to the fields that Child Trends works in. From basic demographic information on children and families, to attitudes about school, community involvement, children’s behavior, and parenting, we depend on high-quality survey data to monitor trends, track progress toward goals, and inform practitioners, policy makers, and advocates. However, survey design is a science in itself. Child Trends has deep experience in helping our clients create, administer, and analyze survey results to inform practice and decision-making, and we know how much thought and planning are required to make them as accurate as possible.

As it has with so many things, [inlinetweet prefix=”” tweeter=”” suffix=”- via @ChildTrends”]the internet has changed the landscape when it comes to surveys[/inlinetweet]. [inlinetweet prefix=”” tweeter=”” suffix=”- via @ChildTrends”]These days, you’re more likely to fill out a survey online than to talk with an interviewer by phone[/inlinetweet]. Even government-sponsored surveys are getting into the act. The next Decennial Census, in 2020, will be the first to count online responses in addition to those collected by mail or in person. There are multiple reasons for this change, including reduced cost, the difficulties of including mobile-phone users in surveys, and harried householders’ desire to limit intrusions into their schedules.

But how reliable are data that are collected this way? After all, in contrast to more traditional modes of survey delivery, there is no specified sampling frame for internet users. (A “sampling frame” specifies one or more characteristics of a finite group—one that can be counted—and provides the basis for drawing a sample that is representative of the whole population of interest.) [inlinetweet prefix=”” tweeter=”” suffix=”- via @ChildTrends”]For online surveys, the only ticket to participate is access to a connected device[/inlinetweet].

A recent report from Pew Research tackles this question. The authors gave an identical 56-question survey to nine different online non-probablilty samples assembled by different vendors. These results were compared with those from a panel of respondents recruited using well-established random-digit-dialing procedures, and who were given a mail-in response option if they lacked internet access. Researchers evaluated the samples using a number of metrics, including comparison with 20 benchmarks based on high-quality government data.

The Pew researchers offer a preliminary conclusion that non-probability surveys (those where any given respondent has an unknown chance of being selected for participation) can, in some circumstances, yield useful—and much less expensive—data. Further study with a greater number of samples will be necessary to substantiate these findings.

[inlinetweet prefix=”” tweeter=”” suffix=”- via @ChildTrends”]Survey vendors call on a variety of techniques to increase the accuracy of online surveys[/inlinetweet] at every stage, including participant recruitment, sample selection, and even decisions about how long a survey remains open. Even so, some biases were consistent across a number of the online samples Pew examined. For example, all of the surveys over-represented people who are more civically and politically engaged; they skewed toward adults without children, and adults who live alone, collect unemployment benefits, and are low-income. Black and Hispanic individuals were poorly represented across the board.

[inlinetweet prefix=”” tweeter=”” suffix=”- via @ChildTrends”]Online surveys are an example of “non-probability” surveys[/inlinetweet]—that is, the probability of a given individual’s participation is unknown. The accuracy of both probability and non-probability surveys is threatened by non-response. When non-response rates are high, researchers must rely on modeling techniques to improve reliability.

The careful researcher—and the conscientious consumer of research findings—will weigh the pros and cons of any given online survey. In particular, details of how participants are recruited, whether special efforts are made to include historically under-represented subgroups, how items are worded, and whether results are weighted to reflect the actual proportions of subgroups within the population are important. Finally, consider the specific topics a survey deals with, the uses to which the information will be put, and other available sources that may contribute to a more rounded picture of the issue. As more and more survey results appear in the weeks leading up to November, it’s wise to assess their methods and validity before drawing broad conclusions.