Running Head: MODERN POLITICAL POLLING 1

Problems and Solutions for Modern Political Polling

Ben Davison

Western Kentucky University

MODERN POLITICAL POLLING 2

Abstract

This paper details various issues that are currently plaguing political polling as a research field

and as a methodology, and offers solutions for many of these issues. Much of the topics

discussed here will refer to the 2016 Presidential Election, which has become infamous within

political polling as many polling agencies and news outlets had forecasts that predicted the

incorrect winner in Hillary Clinton. This paper will also briefly discuss the importance of

statistical literacy and its importance in understanding and interpreting polls.

MODERN POLITICAL POLLING 3

Problems and Solutions for Current Political Polling

On November 8, 2016, the 58th United States Presidential Election took place. It was

between Republican Donald Trump and Democrat Hillary Clinton, and FiveThirtyEight, the

most popular polling aggregator (an agency that combines multiple polls to build a forecast of an

event), released its final update to their 2016 Election Forecast. Nate Silver, founder and editor-

in-chief of FiveThirtyEight, gave Hillary Clinton a 71.4% chance of winning the election (2016).

Of course, Trump would go on to win the election. Immediately, aggregators and polling

agencies (alternatively, “pollsters”) came under fire for calling the election “incorrectly.”

Statistically, this was a strong accusation, but it was glaringly revealing about the lack of

statistical knowledge of these accusers.

Fundamentally, FiveThirtyEight’s final number, a 71.4% chance for Hillary, was just

that: a chance. It would be different if FiveThirtyEight had claimed, “We are 71.4% certain that

Hillary will win.” However, the keyword “chance” indicated that their forecast was widely

uncertain; it was not exactly a prediction. Even though Trump won the election, Silver’s forecast

wasn’t necessarily wrong; Trump still had a 28.6% chance that was quite reasonable (2016).

However, it was likely true that polls had never been so wrong about a presidential election as

2016. Following the election, polling agencies (such as Rasmussen, Quinnipiac, and Gallup) and

aggregators (like FiveThirtyEight and RealClearPolitics) took a hard look at their methodology,

prompting the mainstream media to follow suit and examine the issues regarding political

polling. Though there are several issues with the actual ways that polling is conducted, it is worth

noting that many of the issues regarding polling can directly be attributed to the way it is

interpreted by the general public due to the new accessibility of such polls and forecasts now

found all over the internet; for example, recall the introduction of the accusations directed at
MODERN POLITICAL POLLING 4

Silver and FiveThirtyEight. In the modern world today, political polling is regularly misused and

misinterpreted, often due to imprecise methodology, and is due for a significant overhaul in the

way it is conducted and presented.

Polling Basics and Recent Results

2016 was a rough year for polling, but it was certainly not the first time polls were

inaccurate. Shirani-Mehr et al. estimated that “the average difference between poll results and

election outcomes … is 3.5 percentage points, about twice the error implied by most reported

confidence intervals” (2018). To understand this result, we need to examine some basics in

polling and statistics.

First, let us start with the fundamentals of polling. Polls are designed to estimate public

opinion on an issue. In this paper, we are focusing on political polls, which are used to determine

how the public will vote in an election. The optimal poll would reach every citizen who is set to

vote in an election, but this is simply unfeasible; there is no reliable way to survey such a large

group of people. As a way to estimate the entire public’s opinion, polling agencies will reach out

to a smaller (often random) group of people whom they expect to provide a representative

sample of the general public. We will go into more detail on this later, but, most importantly,

understand that individual polls only survey a small group of people.

Because this group is significantly smaller, the uncertainty of the poll’s accuracy is much

higher; with a smaller size, it is much easier to be inaccurate. Consider the classic probability

example of flipping a coin. We know that heads will turn up about 50% of the time with a large

enough sample, but, if we were naïve enough to try and come up with a result from just 10 flips,

it is very likely that our result could be far from 50%. It is much easier to get all heads from 10

flips rather than 1000 flips. This is a fundamental illustration of the importance of sample size. In
MODERN POLITICAL POLLING 5

polling, our smaller surveyed group will not be near as accurate as a theoretical survey that

reached all possible voters. Taking this uncertainty into account, statisticians will create a

“confidence interval,” which creates a sort of range of where a theoretical “true” result would

land. Say we get heads 8 out of 10 flips. As far as we know, based on our experiment, the coin

has an 80% chance of being heads on every flip. But, as (relatively) smart statisticians, we

recognize how small our sample size is, and acknowledge our confidence interval should be

large to reflect the fact that our result is hardly experimentally sound because we only had 10

trials. Of course, there are mathematical ways to calculate confidence intervals effectively, but

we will just focus on the concept of a confidence interval.

Going back to Shirani-Mehr et al., we can better understand the conclusions. They find

that the average inaccuracy of polls is about 3.5 percentage points, which is twice the error

implied by confidence intervals. To bring all of our new statistical knowledge together, let us

treat this like a coin-flipping example. Say we have several estimations from different scientists

on the percent chance of heads turning up. Since the actual value, as we know, is 50%, imagine

that the average scientist came up with a value of 46.5%. This is off by 3.5 percentage points.

However, if the average confidence interval was [44.8, 48.2] (46.5 ± 1.7, 1.7 is about half of 3.5

from Sherani-Mehr et al.’s conclusion), meaning that the scientist is confident that their value

falls within that range, the average error would be 1.7 (from our average value of 46.5). The

issue with this is that the actual value of 50% is not within our confidence interval; this is a

significant statistical problem. Ultimately, Shirani-Mehr et al. state that, on average, the actual

poll results are outside of our confidence intervals. This is alerting, as our mechanism for

accounting for uncertainty is inaccurate itself. This indicates that polls are becoming more and

more inaccurate.
MODERN POLITICAL POLLING 6

On the surface, polls may seem relatively simple to conduct. However, pollsters must be

concerned with several things in creating such polls. Statistically, they must avoid all types of

biases; in choosing participants, depending on the target demographic, each individual must have

an equal chance of being selected to participate. Failure to accomplish this, that is, letting certain

individuals of a certain characteristic within a demographic have a higher chance at being

selected in the poll, is known as “selection bias.” In practice, for a poll such as a presidential

election poll, this could mean, for example, trying to randomly choose digits to come up with a

phone number to dial and select for participation. The American Association for Public Opinion

Research (AAPOR) is the chief association of professionals regarding surveying in the United

States and provides multiple guidelines on producing scientifically valid polling results. The

technique described above, known as “random digit dialing,” is described by AAPOR as an

example of building a probability-based sample, where individuals are chosen based off of some

sort of random selection process: “The major advantage of probability-based sampling is that we

can calculate how well the findings from the sample represent the total population” (2015). The

best polls will use such a method, as researchers are able to explicitly determine the chance that

an individual is chosen from a population, given that chance is approximately equal among all

individuals.

Current Problems with Polls

Now that the basics of polling have been introduced, including the ways that individuals

are chosen, we can examine the issues behind some of the methods and perceptions regarding

polling. As noted earlier, polling agencies must make heavy compromises in building samples;

they want a large sample, but it is difficult and expensive to reach out to potential voters.

Discussing Gallup’s (a major American polling agency) findings, Bowman notes that “its
MODERN POLITICAL POLLING 7

response rate for its social series in 2017 was 7% on average, down from 28% in 1997. As

Gallup points out, low response rates ‘do not pose a risk to data integrity,’ but they do mean that

conducting good surveys takes more time and is more expensive” (2018). Essentially, agencies

are struggling to find potential respondents; if Gallup reaches out to 100 people, only 7 will

complete their survey, on average. In order to find a large enough sample for a statistically

significant poll, they must now reach out to about 4 times as many people compared to 1997.

One potential reason for this decrease in polling receptibility is public opinion. Following

the infamous 2016 election, it is no secret that the public has a greater distrust in polling as a

whole (likely for good reasons). Bowman references an NPR/PBS Newshour/Marist Poll that

states just “thirty-five percent [of people surveyed] trusted public-opinion polls a great deal or a

good amount” (2018).

In addition to this declining outreach among pollsters, many of their methods have come

under scrutiny recently. Polling agencies have started to use many surveys that can be treated

much like leading questions, designed to elicit a certain answer. Alternatively, many polls have

used certain wording that can dramatically affect responses. In addition, many surveys have tried

to answer bigger questions by using smaller, specific examples of the topic at hand that may

have been big in the news (and thus subject to other factors that are not directly related to the

greater topic at hand). Each of these examples indicate the idea that some polls have become

significantly more subjective and thus are vulnerable to producing inaccurate results.

Regarding the previously-stated example of the 2016 Presidential Election, we can now

attempt to understand the way that poll aggregation works and scrape the surface of recognizing

the causes that led to inaccuracies such as FiveThirtyEight’s forecast. In the October before the

Presidential Election, Patrick Honner of the New York Times published a lesson plan for
MODERN POLITICAL POLLING 8

teachers that focuses on using basic statistics to predict the upcoming election. Honner provides

very elementary approaches to forecasting an election that are very accessible: using historical

results (how individual states have voted in previous elections), using voter turnout data

(estimating what voting participation may look like), using correlated data (considering other

data that may relate to presidential election results, such as Senate races that may coincide with

the presidential election), and, penultimately, combining these methods. Providing an example of

how such aggregation could work, Honner notes, “students might use the latest polling data as

the basis for their predictions, but then adjust those predictions based on Senate or House races

in a few particular states” (2016).

The main takeaway from this should be simple: polls can be used in many ways to predict

an election. FiveThirtyEight uses forms of all of Honner’s methods and more to build their

forecasts. This brings up a bigger issue regarding polling: there is a significant amount of room

for interpreting polls in different ways. By combining several polls to build a forecast, the

margin of error drastically increases. In including more and more sources, there is a higher

chance of making a mistake. Even though sample size grows, including inaccurate polls has the

potential to skew a forecast dramatically.

Nate Silver made an important note about this when discussing the results of the 2016

election in hindsight: “While FiveThirtyEight’s … forecast gave Trump a comparatively

generous 3-in-10 chance of winning the Electoral College, … some other forecasts [showed]

Trump with less than a 1 in 100 shot. Those are radically different forecasts: one model put

Trump’s chances about 30 times higher than another, even though they were basically using the

same data” (2017). Reiterating previous points, aggregators can go several ways with the polling

data they take. Often, they combine various polls by using a “weighted average,” in which
MODERN POLITICAL POLLING 9

certain polls are given more weight, or importance, within an average. For example, in Silver’s

(2016) FiveThirtyEight forecast, a weight of 8.72 is given to a November ABC

News/Washington Post poll, while an early October Fox News poll received a weight of just

0.16. Although FiveThirtyEight used objective and mathematical techniques to determine the

weights, it is far too easy for aggregators to fabricate these weights, allowing them to

subjectively determine which polls are better. This is due to a commonly-found lack of

transparency among poll aggregators. Following the 2016 election, FiveThirtyEight became

dramatically more open about the way their aggregations worked and the way they selected polls

and corresponding weights.

Reflecting on the election, Silver offered 3 main reasons for why Trump won the

election that polls and aggregations did not completely capture: “First, the background

conditions were pretty good for Trump … Second, demographics gave Trump a big advantage in

the Electoral College … Third, voter preferences varied substantially based on news events, and

the news cycle ended on a downturn for Clinton” (2017). This backtracking of sorts from Silver

is a valid defense, but it exposes another issue with polls that is maybe most prevalent today:

they are taken as gospel when they should only be treated as indicators. Polls fail to capture a

stunning amount of factors, as they should: Polls are not designed to provide a complete election

forecast; they simply provide an overview of what the general public is thinking at the time and

nothing more. So many more things affect an election, and, when accounting for the tremendous

variance that polls can create (as mentioned earlier), it should become clear that polls are not

some sort of oracle for an election.

MODERN POLITICAL POLLING 10

Polling Solution Proposals

Now that we have touched on many of the issues associated with political polling, we

propose various solutions that could combat much of them. Fundamentally, in order to have a

greater rapport with the public, polling agencies must ensure that people recognize the value of

participating in such surveys; the decline in response rates for pollsters like Gallup and Pew

Research is simply unacceptable. To achieve this, it may valuable for agencies to educate the

public on how their survey responses could be of great value; a society in which the public

understands their own opinions can operate more cooperatively. Bowman (2018) summarizes

this: “Polls are intended to serve the public and help us better understand the views of

Americans, but many in the industry seem to have lost touch with the people they are supposed

to poll.”

In addition to the public understanding the value of surveying and public opinion, it is

important that they also have a basic understanding of the methodology; this would combat much

of the false perceptions surrounding political polls in recent years. To avoid another 2016

election situation, the public must understand one key piece of election polling: polls are simply

indicators, not catch-all predictions designed to accurately forecast elections. Individually, polls

are not made to forecast. The forecasting for elections is done separately, through aggregators

like FiveThirtyEight. This distinction is very key, even though the blame for the recent

inaccuracies in polling can be attributed to both institutions. Coupled with recognizing the lack

of an exhaustive prediction provided by polls, the general public must also recognize that polling

methodologies explicitly do not capture everything that affects an election. Cohn et al. describes

a variety of reasons that the 2016 Election Forecast was inaccurate by examining which exact

states were inaccurately forecasted by polling averages, and providing explanations for each
MODERN POLITICAL POLLING 11

state: “Mrs. Clinton tended to outperform in big, liberal states with larger Hispanic populations

… Mr. Trump’s gains among white working-class voters were far more important, because those

voters are overrepresented in the most important battleground states” (2016). This is a textbook

example of how polls, and, to a lesser extent, aggregators, are unable to completely capture the

politics of an election. It is next to impossible to predict certain behaviors come Election Day,

and this should be emphasized in public opinion research.

It would also be of tremendous value for the public to be educated on the way that

aggregations work. As noted earlier, FiveThirtyEight has taken the lead in this by becoming

more transparent about their methodologies, but explanations must remain low-level and non-

technical enough for the average reader. Though sufficiently comprehensive for the statistically-

literate quantitative thinker, FiveThirtyEight’s forecasting guide would not work for a general

audience; statistical concepts are referenced that the average reader would not be familiar with.

On the same token, other aggregators must follow suit and dramatically increase transparency

surrounding their methods; too much of public opinion research and polling has become a sort of

black box where data is kept secret and formulas are hidden. It is unrealistic and unfair for

professionals to expect the general public to take their work as truth, especially in today’s

political climate where issues such as “fake news” remain a constant threat to the journalism

utilized by agencies and aggregators.

When shifting focus towards polling agencies, outlets must place a greater emphasis on

statistical significance and confidence (margin of error) reporting. Shirani-Mehr et al. note that

polling inaccuracies have commonly been found outside of reported confidence intervals, which

is troubling from a statistical standpoint. To combat this, a solution is proposed to essentially

widen the margin of error: “Indeed, empirical election-level bias and variance could be directly
MODERN POLITICAL POLLING 12

incorporated into reported margins of error” (2018). The proposal here is not immediately

obvious, but the suggestion is to account for additional factors that aren’t explicitly quantified in

the margin of error, i.e., agencies should make fewer assumptions about the data, and, in public

opinion research, it is important to be safe rather than sorry.

Finally, some action must be taken to place more emphasis on statistical literacy

throughout the country. Many can get away with going through an entire bachelor’s education

without taking a statistical reasoning course, even though statistics is and will continue to be a

vital part of everyday life, particularly in the digital age. Many of the false public perceptions

continually referenced here towards polling can easily be prevented through some basic

statistical training that emphasizes concepts such as hypothesis testing, sample sizes, and

confidence intervals. It is troubling to consider how much of the general population is unfamiliar

with such topics, yet invigorating to consider how much discord could be prevented if it were

easier for the public to find consensus on understanding commonly-found quantitative values.

Conclusion

Ultimately, public opinion research is a field that the public will continue to rely on due

to the variety of services it provides. However, in particular, it is clear that political polling as a

field is struggling due to issues plaguing its work. Chief amongst these issues are mistakes within

their methods, a lack of transparency, and misconceptions that the public is responsible for. On

the contrary, Nate Silver wrote that “while there have been some genuine trouble spots, like

polling in the 2016 presidential primaries, overall there simply hasn’t been a clear trend towards

polls becoming either more accurate or less accurate over time” (2018). Even if the accuracy of

polls is not necessarily under a direct threat, the industry must still recognize the validity of

much of the issues described here, particularly in the lack of transparency provided by agencies
MODERN POLITICAL POLLING 13

and aggregators. Finally, a few proposals were offered here to solve some of these issues, but it

is inevitable that political polling is a field that will be nearly impossible to perfect, due to the

several limitations (particularly in surveying and sample size). Moving forward, polling agencies

must make sure that their work is accurate and generous in reporting error, and aggregators must

ensure transparency in their reporting and objectivity in their methods. For the general public, the

value of statistical literacy cannot be overstated; the best inhibitors of knowledge are those who

understand what the knowledge represents, as well as what it does not represent.
MODERN POLITICAL POLLING 14

References

American Association for Public Opinion Research (2015, December 10). Sampling Methods for

Political Polling. Retrieved from https://www.aapor.org/Education-Resources/Election-

Polling-Resources/Sampling-Methods-for-Political-Polling.aspx

Bowman, K. (2018). The Trouble with Polling. Retrieved from

https://www.nationalaffairs.com/publications/detail/the-trouble-with-polling

Cohn, N., Katz, J., & Quealy, K. (2016, November 13). Putting the Polling Miss of the 2016

Election in Perspective. The New York Times.

Honner, P. (2016, October 5). And the Winner Is ...? Using Statistics to Predict the 2016

Presidential Election. The New York Times.

Shirani-Mehr, H., Rothschild, D., Goel, S., & Gelman, A. (2018). Disentangling Bias and

Variance in Election Polls. Journal of the American Statistical Association,113(522),

607-614.

Silver, N. (2016, November 08). 2016 Election Forecast. Retrieved from

https://projects.fivethirtyeight.com/2016-election-forecast/

Silver, N. (2017, January 19). The Real Story Of 2016. Retrieved from

https://fivethirtyeight.com/features/the-real-story-of-2016/

Silver, N. (2018, May 30). The Polls Are All Right. Retrieved from

https://fivethirtyeight.com/features/the-polls-are-all-right/