Jeff Bezos, George Gallup, and the Definition of Market Research


In his 2018 Letter to Shareholders, the CEO of Amazon, Jeff Bezos, said that “market research doesn’t help.” Bezos was referring to Amazon Echo and making a point about product testing. He said that “If you had gone to a customer in 2013 and said ‘Would you like a black, always-on cylinder in your kitchen about the size of a Pringles can that you can talk to and ask questions, that also turns on your lights and plays music’ I guarantee you they’d have looked at you strangely and said ‘No, thank you.’”  

Is market research worth it?

In the 1940s, Hollywood studio executives wanted to update how they measured audiences. At the time, their methods included “reports from exhibitors, fan mail, or their own intuition,” according to one study of the industry. The systematic surveying of people as consumers — surveys were initially used by social reformers, governments, and psychologists — was transforming how businesses and advertising agencies collected information about people. Naturally, Hollywood wanted a piece of the action.

Between 1939 and 1950, polling pioneer George Gallup was hired to conduct over 5,000 surveys by movie studios and independent producers. Gallup’s methodologies would go on to influence “the casting, narrative structure, and promotional campaigns of films ranging from Gone with the Wind to Walt Disney’s cartoons.” It’s not a stretch to say that the lowly survey has influenced popular culture in a big way.  

Yet one report to movie executives delivered in the 1940s by Gallup’s business expressed concern about how much you can really learn by polling people: “We report the facts,” the report said. “But we insist upon emphasizing that the conclusions to be drawn from them have their limitations.”

Bezos and Gallup are right. But their skepticism is directed at two different things. Bezos is talking about how people struggle to describe hypothetical products. Gallup is saying that when it comes to research that involves people completing questionnaires, you never really know if what they say is true — there’s some guessing involved on both sides.

The answer to the question “Is market research worth it?” is it depends. Are you asking people to imagine something or answer a question? If it’s the latter, getting the right answers is about formulating the right questions. Here, the quality of research is almost entirely dependent on the person writing the survey.

This subtle but powerful difference gets lost in the phrase “market research.” To borrow an analogy author Sam Harris makes to talk about religion, the phrase “market research” is about as ambiguous as the word “sports.” Does chess count as a sport? Do people who practice Thai boxing have anything in common with people who play professional badminton? As Harris says in a blog post, “To speak of ‘sports’ as a generic activity makes it impossible to discuss what athletes actually do, or the physical attributes required to do it.”  

And so it is with market research…  

Image credit: Credit: J. R. Eyerman/Life Magazine/The LIFE Picture Collection/Getty Images

Sam McNerney