Like using a butter knife to tighten a screw
There’s this brief moment after you finish reading a survey question, when your eyes drop to scan your options.
Ideally, you match the response in your head with one of the options.
Sometimes, though, the question sounds awkward, or none of the options fit quite right. You pick one anyway, and it feels forced, like using a butter knife to tighten a screw.
Entrepreneurs obsess over product-market fit: building a product that matches a real need.
I obsess over question-response fit: writing questions that match the way people naturally think or talk.
This might seem backward.
If you write a question with language people already use, it can feel like you’re presuming the very thing you’re trying to measure — a violation of the guideline to stay “neutral” and never “bias” respondents.
But there’s a difference between pushing a thought into someone’s head and pulling one out that was already there.
After running hundreds of surveys – for agencies and Fortune 500 brands – I’ve found that “don’t bias the respondent” is one of the most misunderstood ideas in survey research, and the single biggest reason people get far less value from surveys than they could.
The problem is that the guideline is vague.
It can mean leading someone toward an answer, shaping response distributions, or just providing context. The ambiguity pushes people to eliminate all direction, including the kind that actually helps respondents recall what’s true.
When the early pollsters like George Gallup talked about avoiding bias, they meant something much narrower: don’t smuggle in an opinion the respondent doesn’t actually have (the definition you should worry about).
In trying so hard to remain “neutral,” we default to the rule “never use real language or you’ll screw up the data.”
The result isn’t accurate data.
It’s bad data.
Remember: we’re not running public-opinion polls here. We’re trying to learn how shoppers make decisions so we can write better messaging.
Is that something you need help with?
Consider booking a Survey Roast.
Send me your survey draft, and for $145, I'll make a 15-minute Loom video with copy-and-paste edits and suggestions to improve your survey data quality.
Cheers,
Sam