To ensure the accuracy of a survey, respondents have to answer all questions truthfully. However, people sometimes get bored and rush through the survey. Just to finish the task at hand, other people opt to satisfice instead–choosing an acceptable answer instead of the best one.
That’s why in order to avoid these incidents, people who create surveys add trap questions to filter respondents who are no longer reliable. Also known as red herring questions, these items weed out survey participants who can no longer be expected to provide honest, accurate, or optimal answers.
If you’re looking to add these survey mechanisms to protect the quality of your survey data, here are some do’s and don’ts in creating trap questions.
Do: A red herring for the red herrings
Intentionally placing trap questions in the quiz helps those who conduct the survey. However, it poses a risk for your respondents because, inherently, trap questions may be taken as a show of distrust from your side or, worse, a challenge to their capabilities. Not only will the offended participant be disengaged from the current survey, but they might develop an aversion to the study or to the party conducting it.
To make sure you continue building trust with your audience while maintaining the quality of the data you gather, you have to be strategic with your traps. One way to do this is to place an obvious red herring near the beginning of the survey, or even at the very start. This acts as a screening question that works for both parties. Respondents will be made aware that there are slightly off questions and will be more attentive moving forward. You can also screen out inattentive or uninterested participants whose responses might be unreliable moving forward.
Don’t: Too many trap questions
While there is no set standard for using trap questions, people would agree that one to two of them is more than enough. It is rare for surveyors to use a screening question, with most red herrings appearing only once somewhere in the middle part.
Remember that the purpose of this activity is to gain accurate and meaningful responses from your sample population. Too many trap questions in your survey turn it into a minefield and drastically reduce the usable responses you get to keep. It is important to note that while these kinds of techniques are used to weed out potentially unreliable responses. However, having too many of these items will risk you losing your honest and interested respondents since people choose red herrings by mistake.
Do: Put them in patterns
Good practice in using trap questions is to integrate them into patterns, with survey creators often expecting respondents to break the pattern or risk answering the trap wrong. Generally, some of the giveaway behavior that throws off surveys are related to patterns such as the following:
- Straight lines. Some respondents answer the same point for every survey question. They may either agree or disagree on all questions, and their answers lack variety. This behavior is often associated with loss of interest, regardless of where this straight line appears.
- Acquiescence Response Bias. For others, participants tend to just agree or say yes to a survey, whether it is a customer satisfaction survey or a health assessment one–a tendency technically identified as acquiescence response bias. This is visually similar to straight lines but is specifically characterized by “yes” or “agree” answers.
- Repetitive patterns. Annoying as it may be, some respondents do not take the survey seriously. This often reflects in a variety of other repeating patterns aside from straight lines.
Trap questions are usually inserted in the middle of a series of questions interrelated with each other. This may either refresh the participant’s waning attention or identify which responses are no longer reliable for the purposes of the survey.
Don’t: Declaring that there are “trap questions”
As mentioned above, trust is essential between the surveyors and the respondents. Still, trap questions are essential for most applications, especially where accuracy and/or honesty are crucial. While a disclaimer might sound honest from your end, it might come across as a declaration of mistrust. Also, by nature, traps are meant to be hidden.
Instead, incorporate your trap question as naturally as possible in your survey. There are other ways of ensuring respondent interest, honesty, and reliability as far as surveys are concerned. Some surveys employ personal, conversational phrasing of questions to establish a connection with the respondent, while others offer an incentive or a reward for certain achievements within the activity.
Conclusion
Trap questions are simple yet effective tools that can help ensure that the data you gather is accurate and reliable. The strategies do’s and don’ts listed above can help ensure that you use these questions without alienating or offending the respondents. However, it is important to recognize that a trap question is only one of many tools to create effective survey material. Make sure to assess your questions properly, choose your target population, and host the survey in an accessible, conducive environment.
