As I’ve combed through survey responses over the past 11 years, I’ve learned a thing or two about what to look for, and how to set yourself up for success. Here are a few basic tips that anyone can implement and ensure that their data is foundationally sound.

READ OPEN ENDS

Although a manual and tedious process (especially in large consumer surveys), it is vital to include one or more open end questions in your survey and to review the responses line-by-line. Outside of the typical keyboard smash responses of the worst offenders, you also want to keep an eye out for repeated responses to different questions, stock-responses like “good”, “innovative”, and “I like this”, and even copied and pasted results from running your question through a search engine. Some may even come complete with superscript citations from Wikipedia!

As an added benefit, if you frame these open end questions appropriately, reading through these responses can also provide additional context for your survey’s quantitative results, giving you a peek into how individuals are thinking and feeling about what you are putting in front of them.

OBSCURE WHO YOU’RE LOOKING FOR

The very best bots and professional survey takers will be whoever you want them to be. They may be a CEO by day and a First-line Worker by night, all the while running a Unicorn Ranch and selling commercial real estate on the side – as long as that is who you want to qualify for your survey.

Beyond these more skilled actors, there are still several psychological incentives for a typical survey taker to answer questions you give them in the affirmative, including prestige bias, social response bias, or self-selection bias (more details in a great blog post from Kwik Surveys here).

To avoid letting unqualified respondents guide your company’s strategy, the key is to mask who it is you are looking for by creating qualification questions that treat each choice equally, avoiding questions where the answer you are looking for is obvious.

For example, let’s say that you are looking for Sales professionals who spend at least half of their time remote from a central office location. The most efficient way to gather that information would be to ask it directly – are you a salesperson who spends at least half of your time away from a central office location?  But asking it that way will clearly signal who you are looking for, creating opportunities for unqualified respondents to easily qualify for your survey.

Instead, ask a broader question about their job responsibilities, making sure to include answer choices likely to run counter to the role you are trying to identify. Just be aware that professional survey takers will often select the option that carries the highest status – so if you are looking for a high-ranking executive be extra cautious with the responses you receive.

To determine how much time they spend on the road, include that option in a small list of other activities and ask them to estimate their time spent in each, and you should be able to narrow in on your target without tipping your hand about who you are looking to survey.

TRAP BAD ACTORS

Bots and professional survey takers also demonstrate behaviors that you can use against them. While their strategies are always evolving, they have tendencies toward over-selecting options, answering inconsistently or without reading instructions, and selecting the top-end of every scale you put in front of them.

To combat over-selection, you can include fake or extremely unlikely options. Do they personally own and use a working Iron Man suit? What about a WW2 Era Submarine? While there may theoretically by 3 people in the world who could realistically say yes, what are the chances they are in your survey?

To check for consistency and attention, include answer choices with specific instructions (e.g. select this one, not that one) and similarly structured questions where selecting the same answer choice means the opposite. This will also help weed out those trying to speed through the survey and using the top ends of scales.

As with any QC measure, be careful about not going too far – respondents are busy and setting expectations that make it too hard to give feedback and ultimately turn away otherwise qualified and honest respondents will lead to sub-optimal or even erroneous results.

ALWAYS LOOK AT THE BIG PICTURE

While each of the strategies above can be valuable on their own, they are a starting point to ensuring quality responses in your surveys. They should be used in conjunction with other methods, and always taken in context.

A single lackluster OE or non-intuitive answer may be a tempting reason to throw out a case, but given your respondents’ limited attention span while taking surveys you must consider whether that answer is indicative of a poor respondent, or just fatigue and waning attention.

What you want to avoid at all costs is assuming that you know what the answer should be, and throwing out anyone who disagrees with you – as that can lead to biased results and missing out on discovering unknown unknowns.