Navigating Dishonest Responses in Niche Survey Samples

‍

When surveying low-incidence groups—or groups to which a small percent of the population belongs—researchers often field a survey to a broad group of respondents, and then use a “screener” question to only select the respondents they are looking for. Among honest survey takers, this method will be an effective way to find members of small groups that survey panels cannot identify through other means. However, some members of survey panels may not respond in good faith. Members of lower-quality panels report frequently being screened out of surveys, and as a result they may respond dishonestly to try to qualify for as many surveys as possible to maximize their incentives. This type of behavior, even if relatively uncommon, can seriously damage the quality of your survey data—drawing research conclusions from respondents who do not belong to the target population will lead to highly inaccurate results. 

‍

Imagine a researcher is studying the behaviors of people with food allergies. Only about 9% of adults in the U.S. have food allergies, so the researcher fields a survey to an initial sample of 2,000 respondents using a screener to ask whether or not they fall into this category. And let’s assume that “imposters”— that is, those who do not truly have food allergies, but report having them in order to qualify for the survey—make up only 5% of this group. 

‍

For many of us, our intuition would suggest that if 5% of the respondents are imposters, only about 5% of the final sample would be imposters who do not actually have food allergies. But this intuition underestimates the threat of disingenuous respondents. The chart below shows us why:

‍

If the researcher starts with a sample of 2,000 respondents, 1,900 (95%) of them will take the survey in good faith. And of these 1,900, 171 (9%) will report having a food allergy. But what about the imposters? Only 100 (5%) of the 2,000 will be imposters trying to qualify for the survey no matter what, but all 100 of them will be included. So, in a final sample of 271, over a third (37%) of the respondents are imposters who have been included because of their fraudulent responses. 

‍

At Verasight, we address the challenge of collecting niche samples through a combination of approaches: 

1. Verasight compensates our community for their time spent on all surveys, regardless of their responses. This means that we build relationships with our respondents based on trust and respect, which in turn leads to honest survey taking and the absence of imposters. 
2. Verasight facilitates building specific samples by building extensive profiles of our community members. By collecting a wide range of information about respondents, we can help our customers find the samples they are looking for without relying extensively on screeners.
3. Verified recruitment with external data and publicly available filings. For example, when working with researchers interested in studying voters, we offer to match respondents to the voter file to verify turnout. For sampling of business professionals, Verasight utilizes employment directories, networks, and license filings to identify eligible respondents for survey invitations, instead of recruiting through screener questions to a more general audience. 

‍

For more about the benefits of Verasight’s methodology, get in touch with our team today.

‍