Diverse sampling is key to understanding emerging behaviors.

For decades, probability sampling—such as random digit dialing— has been considered the “gold standard” of survey research. By providing each person with an equal chance of being selected for a survey, the theory suggests, these approaches will result in more accurate measures of beliefs and behaviors. However, with declining response rates to traditional surveys, recent years have shown evidence to the contrary. Non-probability samples—like online panels and convenience samples—have often performed as well or better than probability methods in areas such as election polling.

‍

But how well do these different sampling approaches measure newly emerging behaviors? We addressed this question in research we presented at the 2022 American Association for Public Opinion Research. Our research utilized a collection of surveys measuring self-reported Covid-19 vaccination rates (from the Pew Research Center), which offered us an opportunity to evaluate probability and non-probability samples in the context of a contemporary public health crisis. For each of these 98 polls, we consulted the methodology statement and coded the sampling approach as probability or non-probability. Then, we calculated the “vaccination rate error” by taking the survey’s reported Covid-19 vaccination rate and subtracting it from the official CDC vaccination rate at the time of the survey. The results are shown in the image below:

‍

‍

We find that average vaccination errors for probability and non-probability samples are virtually identical, approximately 2.5 percentage points. However, these errors tend to run in opposite directions: Probability samples tended to overestimate vaccination rates, while non-probability samples instead underestimated. We take these results as evidence that it is important to approach the study of emerging behaviors with a diverse sampling toolkit. Combining probability and non-probability sampling—the approach we use at Verasight—can lead to the most accurate insights by leveraging the strengths of each type of method.

‍

Want to know more about our innovative sampling methods? Get in touch here.

‍