Excerpt: Sense and Nonsense About Surveys

Schuman, Howard. [2002] 2005. "Sense and Nonsense About Surveys". Pp. 21-26 in Understanding Society, 2nd ed., edited by Margaret L. Andersen, Kim Logio, and Howard Taylor. Belmont, CA: Thomson Learning, Inc.

Schuman's article is another on research methods, briefly discussing the importance of sampling in surveys and how a survey is affected by its own form and content.

The 1930s and 1940s saw the development of probability sampling, which "in its simplest form calls for each person in the population to have an equal chance of being selected". He notes the curious fact that regardless of population size, the sample size needed to achieve a certain degree of precision remains the same: around 1000.

As important as sample size is one's sampling method, which considers who is included in the sample and who is missing. If a sample of appropriate size is drawn from a pool of biased individuals, the results of the survey would support the biased ideas of the sample rather than the ideas of the biased and unbiased population as a whole. Schuman notes that even if selected carefully, there is always an indeterminate amount of error due to non-response from participants.

Things as simple as variation in wording or question order can impact survey results. However, if the wording and order are kept constant over time, reliable correlations can be measured.

He also points out that interpretation of survey results must be comparative: "a 75 percent rating of lectures as 'excellent' takes on a quite different meaning depending on whether the average for other lecture courses is 50 percent or 90 percent". Apart from comparing results to other data, it's important to compare the results of different survey researchers with regard to socially sensitive or taboo issues.

Much of this information is common sense, and there's no real argument here. Still, I find beauty in statistical theory. It's not relevant to my sociological theory but is still fun to think about.

Relevance: 2/5 (irrelevant)
Salience: 4/5 (salient)

References:

• George Gallup - as an example. He used quotas to represent different economic strata, and a poll of his sample predicted (correctly) that Roosevelt would win the 1936 election.
• Isaac Newton - cited in agreement. "As Isaac Newton wrote long ago, error is not in the art but in the artificers".