The National Council on
Public Polls (www.ncpp.org),
an association of polling organizations, has prepared
these answers to frequently asked questions about survey research.
Answers To Questions
We
Often Hear From the Public
NATIONAL COUNCIL ON PUBLIC POLLS
1. Why am I or my friends never
included in political polls conducted for the media?
The reason is fairly simple. There are
about 200 million adult or voting-age Americans. But the average poll has a sample size of
1,000 adults. This means that only one person in 200,000 will be included in any one
national or state poll. To put it another way, it would take 200,000 polls with samples of
1,000 for pollsters to get around to all Americans -- and this assumes no one is called
twice.
Of course, national and local media
organizations conduct several polls in one year. The number of national or local media
political polls you see in a single year is about 250, but can vary depending on where you
live. But even 250 polls in a single year means your chance of being interviewed at least
once is still small.
2. Is a larger sample always better than
a smaller sample?
Larger sample are generally more precise,
but sometimes not. The important rule in sampling is not how many poll respondents are
selected but, instead, how they are selected. A reliable sample selects poll respondents
randomly or in a manner that insures that everyone in the area being surveyed has a known
chance of being selected.
One step taken to insure that all members
of a population have a known chance of being selected includes dialing randomly generated
phone numbers to insure that people with unlisted phone numbers are included along with
people with listed numbers. Another is to dial the number again and again when no one
answers to insure that people often away from home are included in the sample. Another
measure is to randomly select a member of the household for interview instead of
interviewing the person who happens to answer the phone.
Non-random samples occur when there is no
chance of including all members of a population. If enough people have no chance of being
included, survey results will be invalid. Here are some examples: Conducting a poll only
between 9 AM and 5 PM, which excludes the opinions of most working people. Another is a
poll based on a sample that selects itself, such as a radio station call-in poll, which
rules out people not caring to respond or not even listening to the station. Other
examples of self-selected samples are poll questions that appear in newspapers, on web
pages, and in magazines, or any poll likely to have a high percentage of members of the
general public not able or not motivated to make the effort to respond. In these cases,
people who choose to respond are likely to differ from others who do not respond.
3. How can a sample of only 800 or 1,200
truly reflect the opinions of 200 million Americans within a few percentage points?
Sampling methods and measures of sample
reliability or precision are derived from a mathematical science called statistics.
Statistics is a subject taught in colleges and some high schools. Text books on the
subject are available in most libraries.
At the root of statistical reliability is
probability; i.e., the odds of obtaining a particular outcome by chance alone. As an
example, the chances of having a coin come up heads in a single toss is 50%. Heads is one
of only two possible outcomes. The chance of getting two heads in two coin tosses is less
because two heads are now only one of four possible outcomes; i.e., a head/head,
head/tail, tail/head and tail/tail. As the number of coin tosses increases, it becomes
increasingly more likely to get outcomes that are either very close to half heads or
exactly half because, as with two coin tosses, there are more ways to get such outcomes.
Sample survey reliability works the same
way -- but on a much larger scale. As in coin tosses, the most likely sample outcome is
the true percentage of whatever it is we are measuring across the total population. Next
most likely are outcomes very close to this true percentage. A statement of potential
margin of error or sample precision reflects this and often appears in poll stories. Using
a sample of 1,000 as an example, the statement could read: the chances are 95% of coming
within +/- 3% of a hypothetical survey conducted among all members of the population. This
means that 95% of all samples that could possibly be drawn will yield an outcome within 3
percentage points of the true percentage among the population.
Keep in mind that estimates of potential
sample error always assume random samples. But even in true random samples, precision can
be compromised by other factors, such as the wording of questions or the order in which
questions were asked.
There is no single ideal sample size.
Samples of any size have some degree of precision. The question is always whether there is
sufficient precision to draw conclusions as determined by statistical formulae.
4. Why do polls often disagree with how
I and my friends feel about things?
It is unlikely that you have a circle of
friends as diverse as a randomly selected sample, described before, a sample of the entire
nation or of the state where you live. This would mean you have friends from all
neighborhoods, of all ages, very wealthy friends and friends with no wealth, who are from
all walks of life and with educational levels ranging from grade school to post-graduate.
No poll has ever shown all people feeling
exactly the same way on one issue. So the next time you see a poll showing only 30% in
agreement with your point of view, remember, although you may not be in the majority, it
still means that 60,000,000 Americans feel the way you do.
5. Why is job approval for elected
officials asked so often in polls?
Job approval is a question asked in polls
for over 50 years, asked about presidents from Harry Truman to Bill Clinton. It is a
measure of potential electoral success for an incumbent president (or state and local
elected officials) and is useful for trend purposes. Incumbents who fail to win approval
from a majority of the public for the job they are doing are very often in trouble on
election day.
Experience shows presidential job approval
has a lot do with how well people think things are going in the country today. High job
approval does not mean the officeholder is necessarily held in high, personal esteem. Job
approval is just one question which, when analyzed in combination with responses to other
questions, gives a good overall picture of how an incumbent is perceived.
6. How accurate are polls?
The NCPP analyzed final presidential
election polls conducted by the national media dating back over 50 years. When compared
with actual election outcomes, average poll error for presidential elections between 1956
and 1996 has been declining. Average poll error on each candidate during this period was
1.9 percentage points.
Important to this analysis of accuracy is
that most of these polls were conducted within days or even hours before election day.
Polls conducted 1-2 weeks before election day or even longer by local newspapers and TV
stations cannot usually be expected to closely match election outcomes. Earlier polls are
intended to monitor the success of campaigns and to identify the issues or events that
will influence voter preferences on election day.
7. Can wording of questions bias poll
results?
How questions in a poll are worded is as
important as sampling procedure in obtaining valid results.
Most professional polling organizations and
their media clients review the wording of questions as carefully as editors would examine
a manuscript before publication. This process usually calls for a review of several drafts
prior to fielding a poll. Questions are checked for balance: Are they worded in a neutral
fashion, without taking sides on an issue? Does the question represent both sides of an
issue fairly? Answer choices read to poll respondents must also be balanced; e.g., approve
or disapprove, favor or oppose.
The order of questions must be logical.
That is, general questions are asked before specific questions. For example, overall job
approval of an incumbent must be asked before specific questions are asked that may remind
respondents about the incumbent’s successes or failures. The same goes for questions
asking respondents what side they take on an issue which may influence a later question
about opinion of a candidate who takes the opposite side.
Questions are written using clear,
unambiguous, concise language to insure that all respondents, regardless of educational
level, understand them. And since most polls are conducted by phone, a writing style
suitable for the ear is often adopted as opposed to a style more suitable for reading.
8. Why should I respond to a poll?
Each year, tens of thousands of Americans
respond to poll interviews on subjects of national interest. They do so because they want
their opinions heard. A poll or survey is an unbiased way for people to make their views
known to each other, to their government, to businesses, to educators, and many other
institutions. This is one way for average Americans to add their voices to the debate over
important issues of our day. It is our sense that people want to be included in polls.
Polling is only a part of a $3 billion U.S.
survey research industry that affects your life in many ways. Medicines in your cabinet
became available only after they had been tested for effectiveness and safety, tested
among samples of patients. Manufacturers use sampling methods to guard against producing
defective products. Businesses conduct market research by sampling consumer opinions to
help develop products people want and avoid costly mistakes. Even the Bureau of the Census
only interviews everyone in the nation once every 10 years. The Census Bureau along with
other government agencies also conduct sample surveys to obtain social and economic data
we see in the news.
One thing legitimate survey researchers
will never do is try to sell you something. That is called telemarketing. According to
direct marketing industry estimates, there are now 2 million telemarketers who call the
general public to sell products and services. The number of survey research interviewers
who conduct telephone interviews is probably only a small fraction of 2 million. |
