PollingReport.com

Home ] Up ] Perot, Bush, Clinton ] Withstanding the Wave ] [ Rise of Robopolling ] Marriage and the Courts ] Delegates and the Democrats ] How to Forecast an Election ] Case for Online Polls ] Presidency on Life Support ] Iowa Caucuses ] 2000: Clinton's Coattails ] NAES 2000 ] Support for Clinton ] Sampling Error ] Incumbent Rule ] Focus Groups ] Internet Polling ] Questions About Polling ]


 



Mark DiCamillo is director of The Field Poll.

This article appeared in the Nov. 15, 2010, edition of The Polling Report.

 


 


The Rise of Robopolling in California
in 2010 and Its Implications

by Mark DiCamillo



When comparing the polling in California this year to previous years, two things stand out. The first is the sheer number of pre-election polls conducted and reported. When reviewing only polls conducted in California’s general election races for governor and U.S. Senate, by my count there were at least 75 different statewide public polls completed by 14 different polling organizations. And, this doesn’t even include the many private polls conducted in each contest for the various political candidates and campaigns.

The other thing that jumps out is that more of the polls than any year previously were robopolls, also referred to as Interactive Voice Response (IVR) polls.

By my count nearly half of all of the statewide public polls reported in California’s general election in the governor and Senate races were robopolls. If you were to also include polls conducted in local election contests across the state, robopolls constituted the majority of all public polls in California in this year’s general election.

Because robopolls are now so prevalent, it is more important than ever for the media and the public to understand just how these polls differ from traditional telephone polls, especially those conducted by the state’s three leading public polls: The Field Poll, the Public Policy Institute of California, and The Los Angeles Times/USC Poll.


Comparing the Methodologies of Robopolls to Traditional Telephone Polls
The basic survey approach of robopolls is to contact people by telephone using the recorded voice of a professional announcer. The announcer instructs those answering the phone to use the keypad on their phones to answer their poll questions. Traditional polls use live interviewers to call voters who ask each question directly.

Because most of the costs of conducting a traditional telephone survey are derived from the time spent and wages paid to telephone interviewers and their supervisors when carrying out data collection, robopolls are much cheaper to conduct than traditional telephone surveys since there are essentially no interviewer costs associated with conducting these polls.

This is one of the reasons why they are now so prolific. They can be conducted at a fraction of the costs of conducting a traditional phone survey.

However, other than cost, robopolls differ from traditional telephone surveys in a number of important ways.


(1) Short polling period, no callbacks.
Most robopolls are conducted very quickly, over a one day period. They typically make only one attempt to reach a voter at each number dialed. If no one answers the phone they do not make callbacks to that number but simply replace it with a new telephone listing.

By definition, this means that robopolls have significantly lower response rates than traditional polls and are polling only those segments of the voting public that are the easiest to reach.

Contrast this to a Field, PPIC or Times/USC poll which is typically conducted over a one-week period and which makes up to six to eight different attempts at each usable number to try to bring voters into the sample. While this is more costly and time consuming, it produces samples that more closely capture the varied demography of California’s voters -- working and non-working, old and young, white non-Hispanic and ethnic, those living alone and those living in multi-family households.

(2) Limited knowledge about who is actually answering their questions.
Robopolls make calls from a random digit dial sample of all possible residential landline telephone numbers within the political jurisdiction they are polling. The recorded announcer instructs the person answering to tell them if they are a registered voter, leaving this important selection criterion totally in the hands of the respondent.

The Field Poll and the Times/USC Poll sample voters off of lists derived from the state’s official voter registration rolls, as did virtually every private poll conducted for a political campaign in California in 2010. This gives the poll a number of advantages. First, it enables interviewers to ask to speak to a specific individual by name and if that individual is not available, the interviewer can make an appointment to call back that voter at a later time. Also, because the sample of names is derived from lists of known voters, we know by definition that the person we are seeking is indeed a registered voter. Working off a voter list also provides the pollster with the voter’s actual party registration, as well as his or her frequency of voting in past elections, since this information is contained on the official voting records. This information can also be used to ensure that the sample is aligned properly to the state’s actual party registration and in identifying which voters are most likely to vote.

(3) Exclusion of cell phones.
By law, the automated dialing devices used by the robopolls are not allowed to call cell phones. Traditional telephone polls routinely dial cell phones by hand to include them into their samples. Since more than 20% of all California voters are now cell phone-only households and cannot be reached when dialing a random sample of landline phone listings, most robopollsters are systematically excluding these voters from their samples.

(4) Language limitations.
To my knowledge, the pre-recorded messages of most robopolls are in English only. This excludes from their samples the additional set of voters who do not understand spoken English. By contrast, Field, PPIC and the Times/USC polls routinely conduct all of their statewide polls in English and Spanish. In addition, Field’s final pre-election poll this year was extended further to include four other Asian languages and dialects: Cantonese, Mandarin, Korean and Vietnamese.

We estimate that 7%-10% of all registered voters in California would either prefer or require non-English language interviewing when completing a telephone survey, so this portion of the state’s fast-growing ethnic voters is under-represented by the robopolls.

(5) The need to construct a model and apply larger weighting adjustments.
Each of these factors means that the quality of the raw unadjusted survey data derived from robopolls is significantly lower than that of traditional telephone polls, like those conducted by Field, PPIC and the Times/USC.

In their methodological descriptions robopollsters admit that women are much more likely to participate in their surveys than men, and that older voters are included in their samples in far greater numbers than young or middle age voters. Because their initial data are less representative, robopolls need to make fairly major adjustments to their raw data to bring their samples into balance with the characteristics of the larger voting population.

By contrast, the unadjusted raw data obtained by traditional telephone pollsters more closely reflect the actual population of voters they are polling. While The Field Poll does make weighting adjustments to its samples, the adjustments tend to be small and have a modest impact on the overall statewide findings.

For example, The Field Poll’s final pre-election poll this year showed Democrats Jerry Brown and Barbara Boxer ahead of their Republican opponents in the races for governor and U.S. Senate by eight to ten percentage points in both our unweighted and weighted samples. The main impact of the weighting or sample adjustments was to align the various subgroups to known characteristics of the voter population. Importantly, they did not have much impact or significantly alter the overall statewide preference distributions initially found in the survey.

Despite their sampling drawbacks, the better robopollsters are able to transform the survey information they obtain into reasonable pre-election poll estimates by developing a sophisticated model of the probable electorate and adjusting their sample to conform to its characteristics. Because of this, I view the better robopollsters more as skilled modelers of the electorate than as high quality survey researchers.

But because of their need to construct models to determine the overall shape of the probable electorate rather than rely on actual survey data or information about each respondent’s voting record to determine this, the modeling itself can create potential problems.

For example, the determination of how many Democrats and Republicans to include in a sample is closely tied to voting preferences. When making this determination the robopollster takes great liberties in deciding who is ahead and by how much, since even a slight change in the partisan distribution of the sample will affect the preference distributions in most election contests.

This is perhaps the most worrisome aspect of the robopoll method, since it confers upon robopollsters greater latitude in influencing the outcomes of their poll measures, and risks introducing systematic biases into their polls.

Some robopollsters admit to taking into account a state’s voting history, national trends and recent polling to construct their partisan weighting targets. This means that the proportions of Democrats and Republicans allocated to their sample are derived from subjective judgments about the historical and prevailing political conditions in a given state and from other polls already conducted in that political jurisdiction.

It would be revealing to be able to compare a robopoll’s unadjusted and adjusted poll distributions in their pre-election preference measures. I suspect that if this information were available, it would reveal wide differences between the two estimates.

Because robopolls make subjective judgments when establishing their estimates of the composition of the likely electorate, this method can easily produce an entire array of different possible survey results. It is left to the robopollster to choose which result or political reality fits their expectations at that moment in time. This is not only dangerous, it has the long-term effect of undermining public confidence in the objectivity of the entire public opinion polling process.


Concerns About the Future of Polling
As more pre-election polls employ the robopolling method, my fear is that they will crowd out the other higher quality polls that are being conducted, leaving the media and the public with a sometimes confusing array of pre-election poll estimates to sort through.

This is not to say that better robopollsters are manipulating their poll results for their own ends. Some are trying to make up for the deficiencies in their initial survey samples. For example, at least one robopollster extended their data collection over a longer three-day period and experimented with the use of live interviewers to call separate samples of cell phone listings in an attempt to fill the gap of voters only reachable by cell phone.

Yet the other problems inherent in their survey approach remain. This is why I continue to view the results of most of them cautiously.


A Final Aside: Polling on Proposition 19
One other controversy between robopolls and traditional telephone polls surfaced this year in California during the Proposition 19 marijuana legalization initiative campaign.

When polling this year on Prop. 19, the traditional telephone pollsters fielded a number of inquiries from reporters and others questioning the reliability of live interviewer telephone polls conducted on a controversial topic like marijuana. The theory they presented was that because robopollsters avoid direct human interactions when conducting their polls, voters felt less constrained about admitting their true opinions on Prop. 19.

Most of the literature regarding interviewer effects on research on sensitive topics, like marijuana, relates to people being asked about their own personal behaviors that might be embarrassing or socially undesirable. This, in my opinion, doesn’t apply when polling on a policy issue like Prop. 19, which simply asks voters their opinions about an initiative to legalize marijuana’s sale and use.

At the time, I challenged those questioning the accuracy of live interviewer polls on the topic to revisit the issue after the election. Well, the results are in and the live interviewer polls like Field, PPIC and the Times/USC polls were generally closer to the final vote on Prop. 19 than the robopolls.

In their final pre-election surveys the state’s three leading traditional telephone polls showed Prop. 19 trailing by an average of 8 percentage points. By contrast, the average of the two final pre-election robopolls conducted in California showed Prop. 19 trailing by just 4.5 percentage points. According to the California Secretary of State, with nearly nine million votes counted and more than one million votes yet to be counted, California voters were rejecting Prop. 19 by eight percentage points, 54% to 46%.

I hope this puts that theory to rest.
 


Copyright © 2010 POLLING REPORT, INC.

Because robopolls are now so prevalent, it is more important than ever for the media and the public to understand just how these polls differ from traditional telephone polls . . . .

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


 

 

 

 

 

By law, the automated dialing devices used by the robopolls are not allowed to call cell phones.

 

 

 

 

 

 


 

 

 

 

 

 

. . . The Field Poll’s final pre-election poll this year showed Democrats Jerry Brown and Barbara Boxer ahead of their Republican opponents in the races for governor and U.S. Senate by eight to ten percentage points in both our unweighted and weighted samples.

 

 

 

 

 

 


 

 

 

 

 

 

 

 

The theory they presented was that because robopollsters avoid direct human interactions when conducting their polls, voters felt less constrained about admitting their true opinions on Prop. 19.

 

 

HOME | TABLE OF CONTENTS | SEARCH THE SITE

Copyright © 2014 POLLING REPORT, INC., and polling/sponsoring organizations