THE TEN COMMANDMENTS OF POLLING
                         by Ken Blake
                       UNC-CHAPEL HILL 1996

1.  Thou shalt know how the people interviewed for the poll were chosen. The best polls choose people at random. There are many different schemes for doing so. For example, a pollster can program a computer to generate telephone numbers at random. Another strategy involves putting people's names on a list in some non-systematic fashion and then choosing every sixth, tenth or "whateverth" one starting from some random point on the list. The basic rule is that everyone whom the poll purports to represent should have an exactly equal chance of being interviewed. The results of polls based on non-random samples -- the "dial-in" polls so popular with television newscasts, for example -- are generalizable to no one except the people who were interviewed.

2.   Thou shalt know the poll's response rate. Here's how to calculate a poll's response rate using any calculator. 1. Enter the number of people interviewed for the poll. 2. Press the division key. 3. Enter the number of people the poll attempted to interview (for example, the number of questionnaires mailed out, or the total number of working, residential telephone numbers dialed in a poll claiming to represent residents of some population). 4. Press the "equal" key. 5. The resulting number should be .66 or higher. In other words, 66 percent or more of the people asked to take part in the poll should agree to do so. The lower the response rate, the great the risk that those who declined to be interviewed belong to some significant subgroup whose attitudes and opinions the poll will fail to represent.

3.   Thou shalt know the poll's error margin. Here's how to calculate a poll's error margin using any calculator. 1. Enter the number .25. 2. Press the division key. 3. Enter the number of people interviewed for the poll. 4. Press the "equal" key. 5. Press the "square root" key. 6. Press the multiplication key. 7. Enter the number 1.96. 8. Press the "equal" key. 9. Press the multiplication key. 10. Enter the number 100. 11. Press the "equal" key. The resulting number is the poll's accuracy at the 95 percent level of confidence. For example, a result of 4.2 means that if 60 percent of the people interviewed for the survey said they preferred chocolate ice cream over vanilla, then one can be 95 percent confident that the actual percentage of people favoring chocolate over vanilla in the population the poll purports to represent lies somewhere between 55.8 (that's 60 minus 4.2) and 64.2 (which is 60 plus 4.2). What's a good error margin? It depends. An error margin of 4.2 will let you predict the winner of an election if the poll shows one of two candidates has 80 percent of the vote. Subtracting the error margin from 80 percent reveals that, at the very worst, the candidate has 75.8 percent of the vote and is clearly still ahead. If, however, the poll shows one of two candidates with 53 percent of the vote, then you can't predict a winner (without using some really hairy
statistics that you'll need a computer and some fancy software to calculate) because the candidate could have anywhere from 57.2 percent of the vote to only 48.8 percent of the vote.

4.   Thou shalt know the poll's confidence level. The 95 percent confidence level is the most common. Confidence levels exceeding 95 percent are even better. Confidence levels less than 95 percent should be increasingly mistrusted as they get further from 95 percent.

5.   Thou shalt recalculate the error margin when drawing conclusions about a subgroup in a poll's sample. If 600 randomly chosen North Carolinians are interviewed for a poll, the poll's error margin at the 95 percent level of confidence is plus-or- minus four percent. But conclusions about the, say, 312 people in the sample who are female have an error margin of plus-or-minus 5.5 percent at the 95 percent confidence level. A pollster who wants an error margin of plus-or-minus four percent at the 95 percent level of confidence when drawing conclusions about North Carolina females needs to talk to 600 randomly selected North Carolina females, not 312.

6.   Thou shalt make a big deal out of a crosstabulation only if the difference is significant. A crosstabulation compares how people in different subgroups answered a question. For example, a poll might find that 60 percent of the women interviewed plan to vote for Candidate A compared to only 55 percent of the men interviewed. So, Candidate A enjoys stronger support among women in the general population than among men in the general population, right? Not necessarily. As a rule of thumb, it takes a difference of 10 percentage points or more between subgroups in a random sample to accurately suggest that a similar difference exists in the general population from which the sample was drawn. Ideally, you should get a statistician to calculate something for you called a Chi Square. Or, you can pick up just about any basic statistics book and figure out how to calculate a Chi Square on your own. It's too involved to detail here, but you can do it with any calculator and the table of probabilities in the back of the statistics book.

7.   Thou shalt generalize a poll's results only to the population sampled. A poll based on a sample of UNC-Chapel Hill students age 18 to 21, for example, does not necessarily represent the views of all people age 18 to 21. Nor does a poll based on a sample of North Carolina state residents necessarily represent the views of all Americans.

8.   Thou shalt know who paid for the poll, and who conducted it. Exercise a little healthy skepticism if whoever paid for the poll has a vested interest in its outcome, and if whoever conducted it doesn't have a solid reputation for telling the truth.

9.   Thou shalt know the wording of the question. Leading questions are easy to write. For example, "You certainly don't plan to vote for putting that crook, John Smith, back into office, do you?" might tend to yield results unfavorable to John Smith, whatever his actual character might be. Balanced questions present all possible answers as equally attractive. For example, "Do you plan to vote for John Smith, or for Jane Doe." Also try to find out the wording of previous questions. The answer to one question can be influenced by questions asked earlier in the poll. For example, a respondent asked about support for the death penalty after being asked several questions about the threat of crime might answer differently than a respondent asked about support for the death penalty during a poll that made no mention of crime.

10.  Thou shalt consider the ability of the people interviewed to give an informed, well-considered answer. There's not point in doing a poll on whether Newt Gingrich wears boxers or briefs. Only he knows, and he's not telling. What everyone thinks he's wearing isn't going to alter the reality of what he is wearing. If there are any appropriate ways to investigate the question, polling is certainly not among them. Similarly, ordinary people should not be asked whether they approve or disapprove of U.S.
policy on Argentinian beef imports. Most have no idea what the policy is, or if there even is one. Even if the pollster patiently explains the policy, the resulting answer is likely to be a snap judgment prone to reverse itself the next day. The best  polls probe not only a respondent's opinion but also the level of  deliberation and conviction associated with the opinion.