My $538.17 typo
Last week a bad question cost me $538.17.
I was writing a screener for a survey about engagement rings:
Do you plan to propose or get proposed to in the next 12–24 months?
450 respondents answered the question. The data came back. Everything looked fine.
Then I was looking up data on marriages per year, and the "per year" part snagged in my brain.
Shit. Did I ask about people getting engaged this year?
I pulled up my screener. "In the next 12–24 months."
Fuck. I needed people planning to propose in the next 12 months.
I fixed the question and re-ran the survey.
$538.17 out of my own pocket.
Later that night, I mentioned my mistake to my wife. "12–24 months is confusing," she said. "Why are you asking about 2027?"
She heard it in five seconds. I missed it for 450 respondents.
There’s an old Jerry Seinfeld joke that public speaking is more terrifying than death—so at a funeral, most people would rather be in the coffin than giving the eulogy.
Seinfeld of all people gets the upside of that fear. Comedians don't try new material in empty rooms. They need the laughter and dead air to tell them what to keep and what to cut.
Surveys don’t work that way. You’re communicating to hundreds of people without facing them. You’re spared the fear. And the feedback.
I don’t think that’s a good trade. Being wrong and not knowing it is worse.
Back in the 1930s George Gallup instructed interviewers to report back on questions that confused people. Today everything is online—which is great for scale, but there's no substitute for seeing someone squint at a question that made perfect sense in your head.
You need those reactions just like a comic needs a crowd.
My advice:
If you have a spouse, read your questions out loud to them. If something sounds weird coming out of your month in front of that person, it will sound weird to strangers online.
If you’re all alone and you hate your colleagues, read your questions out loud to yourself. Step away from your computer first. On a screen, jargon blends in. Out loud, it stands out.
If none of those work for you, Book a Roast.
They’re designed to get the feedback Gallup’s interviewers reported back with.
They are not about methodological nerd tips–although I’ll give you that stuff if we need it.
I’m listening to your questions the way a respondent would. If a question is confusing, I’ll tell you.
Send me your survey draft. I’ll record a 15-minute Loom video with clear, copy-and-paste edits and practical suggestions to make your questions easier for real people to answer.
If you want to avoid paying $538.17 to learn something in hindsight, book one.
Cheers,
Sam