Pollsters face another reckoning this year, but the reasons could differ from 2016 (original) (raw)
Poll workers wait in line to grab breakfast prior to the polls opening at the Registrar of Voters on the day of the Presidential election in San Diego, California, November 3, 2020.
Mike Blake | Reuters
Once again, public opinion polls missed the mark on the U.S. presidential election.
The polls largely indicated a Democratic sweep with former Vice President Joe Biden up an average of 7.4 percentage points ahead of President Donald Trump in national polls, according to NBC News' polling average. Polls indicated Biden could win Wisconsin at 6.7 percentage points ahead of Trump, according to a RealClearPolitics average, although preliminary election results show the candidates are less than a percentage point apart.
Polls on average also predicted Biden would win Florida by nearly a full percentage point, according to RCP. Instead, Trump won the state by more than three percentage points, according to NBC News projections.
But the reasons for the disparity could vary from those in 2016, polling experts told CNBC. Pollsters faced a public reckoning after failing to predict former Secretary of State Hillary Clinton's defeat by Trump. Since then, the industry made changes seeking to diminish the error in their methods.
"When polls go bad, when polls mislead, they do so in different ways," said W. Joseph Campbell, a professor of communication at American University and author of "Lost in a Gallup: Polling Failure in U.S. Presidential Elections."
"No two polling failures are quite alike and I think that held in this year," Campbell said. "2020 is not the same kind of polling surprise as it was in 2016."
Polling groups took stock of the failures in 2016 and tried to adjust, as they always do. A post-mortem by the American Association for Public Opinion Research (AAPOR) found that polls ahead of the 2016 election mainly underestimated support for Trump in places like Michigan, Pennsylvania and Wisconsin, which Trump flipped from blue to red that year.
AAPOR found that failing to weight results based on education level was a likely source of error in the polls, among others. Education level was later found to be highly correlated with voters' preferences in key states, according to the report, though pollsters still disagree on the importance of this factor in accounting for the error.
Overall, AAPOR found that national polls in 2016 were not all that far off, showing a 3 percentage point lead for Clinton in the popular vote, which she ultimately won by 2 percentage points. Even many state-level polls "showed a competitive, uncertain contest," AAPOR found, and said that on average, they "indicated that Trump was one state away from winning the election."
This year, many polls appear to have been directionally correct based on preliminary vote counts which exclude many outstanding and mailed ballots. Charles Franklin, who directs the Marquette Law School Poll, calculated that more than 90% of polls got the winner right in states that had been called as of early Friday afternoon.
But on average, he said of the races called so far, the polls overestimated Biden's lead by 5.8 percentage points. He said he he did not have data on six states not yet called as well as Mississippi or Massachusetts.
Franklin said the error this year seemed to be more broad.
"In 2020, rather than having that error focused in just a handful of states, this year really it looks like it was quite widespread," he said. "I can't find any state where the polling error was in Trump's favor."
While it's too early to fully assess what went wrong this year, especially with many still votes still outstanding, pollsters have some early questions. Those range from how record turnout could impact the numbers to how Trump himself might have.
"There's no single answer to why the polls surprised us in 2020," Campbell said.
Where 2020 polls could have gone wrong
Combination picture of Democratic U.S. presidential nominee Joe Biden and U.S. President Donald Trump speaking about the early results of the 2020 U.S. presidential election, U.S. November 4, 2020.
Reuters
Polling experts stressed that it's still far too early to assess where exactly polls went astray this year. Without all of the votes fully counted, it's impossible to say just how far off the polls will end up being and it's possible final results could diminish the size of those differences.
AAPOR warned in a statement "that hasty conclusions based on incomplete returns may be misleading."
But pollsters who spoke with CNBC detailed several areas they are keen to look into once it comes time to assess this year's poll performance.
One factor could be the record turnout this year, which marked the highest rate among eligible voters since 1900 at nearly 67%, according to the U.S. Elections Project.
It will be important to figure out which groups of voters turned out in higher numbers than in previous years. For example, perhaps voters who favored Trump turned out in greater numbers on Election Day than accounted for, or maybe more Cuban-Americans in Miami registered or cast ballots compared to previous years.
Another area pollsters hope to better understand is who is being left out of polls and how they might better be able to reach those people.
"The biggest threat to the validity of polling is if a particular group of voters systematically won't talk to the pollsters because I can't learn anything about the preference of someone that just wont talk to me," Franklin said. "This is different from the shy Trump voter idea that someone agrees to do an interview but then is reluctant to report that they are voting for Trump."
Emerson College Polling Director Spencer Kimball has experimented with different ways of reaching voters more recently as landlines become more rare and difficult to reach. This year he's used methods including online surveys and text messages. Kimball said that expanding the universe of people pollsters can reach makes it more likely they will capture a wider array of voters.
Still, some groups of voters can be particularly hard to get a hold of. Patrick Murray, director of the Monmouth University Polling Institute, said he specifically looked for differences by ethnicity among the Hispanic population in Florida because political leanings within that group can vary widely. But Murray, who also serves as an exit poll analyst for NBC, said his polls' assessment of southeast Florida were particularly off, possibly because much of the population moves around frequently or is Cuban-American, a group several pollsters said is difficult to get a hold of.
Kimball said his "biggest mistake" in his Florida polling was combining Miami-Dade County with Broward County. Miami-Dade did not break as heavily for Biden, potentially due to the large Cuban population there that may have favored Trump to a greater degree.
But one method Kimball stands by is his decision to factor in which candidate respondents voted for in the last election. Kimball said some pollsters don't believe in using the past vote to predict the future, but he said he believed the electorate "stayed pretty consistent and grew on both sides" instead of falling away from Trump.
Because of that, Kimball said he initially doubted some of his results, which were more conservative than other outlets. Emerson's last national poll predicted a Biden win by 5 percentage points, 2.2 percentage points below the RCP Average.
'Donald Trump effect'
President Donald Trump is reflected as he departs after speaking about the 2020 U.S. presidential election results in the Brady Press Briefing Room at the White House in Washington, November 5, 2020.
Carlos Barria | Reuters
One consistent factor between 2016 and 2020 polls was the name at the top of the Republican ticket.
"That might be something that we just are learning right now, that there's a Donald Trump effect in polling," Murray said.
After 2016, pollsters debated the possibility that "shy Trump voters" failed to disclose their preference for Trump. AAPOR found some evidence to support this theory in its post-mortem, but said "A number of other tests for the Shy Trump theory yielded no evidence to support it."
Franklin said that rather than being shy, Trump voters may simply be less likely to respond to polls in part due to Trump's own attacks on them. That could make their views harder to track.
"Every president has criticized the polls, but no one has systematically attacked them in the same way that the president has done," Franklin said.
He also said his polls in Wisconsin this year seemed to support the idea that Trump voters are harder to capture in the data than Biden supporters. Marquette Law School's final Wisconsin poll showed Biden was the preferred choice of 48% of likely voters with a +/- 4.4% margin of error. That's not far off from the 49.4% of the vote Biden has gained based on Friday's tally.
But the poll only showed 43% of likely voters favoring Trump with the same margin of error. Trump has actually pulled 48.8% of the vote in Wisconsin as of Friday's tally.
Still, Franklin said, the amount by which his polls underestimated Trump this time was less than in 2016, though "it does still look like we're missing some Trump voters for whatever reason." He said this year, his poll assigned undecided voters to a candidate if they were determined to be favorable to that person. That ended up boosting percentages for both candidates fairly evenly, he said.
Kimball said that it's possible Trump could be a "political anomaly" for pollsters.
"We didn't see those errors in 2018 and we don't see those errors in primary elections," he said. "But we sure saw them when Donald Trump is on the ticket."
WATCH: How to determine if polling data is trustworthy
