‘I Don’t Have a Happy Ending’: A Pollster on What Went Wrong

Hi. Welcome to On Politics, your guide to the day in national politics. I’m Lisa Lerer, your host.

Sign up here to get On Politics in your inbox every weekday.

Over the past few days, there’s been much consternation over this year’s polling of the presidential race. How did we end up with a close election in key swing states after months of surveys that suggested there could be a Democratic landslide?

It’s a familiar crisis: After the 2016 election, pollsters faced a chorus of recrimination about missing the mark and conducted a major analysis of what went wrong.

Charles Franklin, the director of Wisconsin’s best-known political survey, the Marquette University Law School Poll, helped compile that 2016 report. He’s been involved with political polling since 1980 and has been polling his swing state for more than a dozen years. This cycle, his polls missed the mark again — though not by as large a gap as many of his competitors (including the New York Times/Siena College polls).

I talked to Dr. Franklin about the crisis of confidence in polling, the challenge in identifying Trump supporters and why the surveys got it wrong, again. As usual, our conversation has been edited and condensed.

You and I spoke last year about what the polling got wrong in the Midwest in 2016. Well, here we are again, with another poll-defying outcome.

The last Times/Siena poll of Wisconsin, conducted just days before the election, gave Joe Biden an 11-point lead in the state. Your survey had Mr. Biden leading by five points. He won Wisconsin by less than a point. What happened?

I want to begin with the obvious, which is that I know about my data, but I only know what I know from public sources about everybody else’s.

What we see in our data is that we have been getting the Democratic vote amazingly close. Almost all of the error in our 2016 poll came from a substantial understatement of Trump’s vote. It was clear that we were understating Trump’s vote in the suburbs, especially in the Milwaukee suburbs and to a lesser extent the Green Bay suburbs.

Flash forward to this past week. I was worried that we had Trump voters lurking that we didn’t find in 2016. So what we did this time is, for voters who said they were undecided or declined to say how they would vote, we used their favorable or unfavorable views of the candidates to allocate them to either a Biden or a Trump vote. If you were favorable to Biden but not favorable to Trump, we allocated that person as a Biden voter, and vice versa. (Three percent of respondents were still unallocated, most of whom were unfavorable to both.)

After doing that, we did a lot better this time than four years ago. We were off on the winner last time — we had Clinton up by six points and Trump won by 0.77 points. So a seven-point error for us last time. It’s a four-point error this time. That’s better.

We have the right winner but the same phenomena occurred — accurate on the Democrats, understating Trump.

That seems to be a trend in the polls again this year. Why did pollsters have so much trouble finding Trump supporters?

I don’t think this is the “shy Trump” voter in the way we’ve understood it, as people not wanting to admit they’re voting for him. Lots of effort has gone into finding evidence of that and it just doesn’t seem to exist.

I’m more inclined to think we’re seeing a phenomenon of some fairly small segment — 3 or 4 percent, maybe, of Trump supporters — who systematically decline to do surveys altogether. That would fit with the notion that some segment of his supporters are pretty anti-press, anti-polls and in a lot of ways anti-conventional political engagement. But they may also be people who are not, in fact, strongly identified with the Republican Party. My hypothesis going forward is to search for the evidence that there is this small but critical segment of the electorate.

I’m going to hold out one other possibility, which I don’t have the data for yet. And that is the possibility that there was a surge in Election Day turnout that we didn’t catch in the polling, given the partisan imbalance between early and Election Day voting.

So if Trump is no longer on the ballot, will polling become more accurate? Or has it simply become more difficult to survey Republicans? If that’s true, it would really complicate our ability to get a sense of the electorate.

The worry that we have is that survey nonresponse might become correlated with partisanship. One of the questions that I always get is: “I never pick up the phone if I don’t know the number. How can you possibly do surveys?” And my routine answer is that Republicans and Democrats alike hate telemarketing and scam phone calls. That has been the great blessing of the political polling industry. While our response rates may be lower, it has been an equal opportunity.

Over the last four years of the Trump presidency, we have not seen any trend of a falling Republican percentage of the electorate, which you would expect to see if Republicans were now systematically as group refusing to do interviews. So I don’t think this has spread to the broader Republican electorate. However, it’s certainly a worry, because if it should spread, it will make Republicans’ standing in the electorate look worse because we’re missing Republicans.

The Times conducted a large number of polls this cycle that had many of the same problems. We’re not blameless either. Is there a better way that media organizations should be using polling?

This is not sucking up, but I’m really impressed with the job The Times and CNN have done over the last couple of years. The thing that’s puzzling is that every serious polling operation looked for the hints that the polling problems of 2016 were still there. And it’s really worrisome that we failed to find those hints.

Alternatively, it may be that we were looking for hints of problems different from the ones we actually experienced. To quote a former defense secretary, it’s always the unknown unknowns rather than the known unknowns.

What I think will be very hard to convey to anybody outside the polling world is that this is a result despite widespread efforts to address the problems of 2016, not because of negligence and ignoring problems that we knew we had.

In a way that’s more disturbing. Despite trying to find sources of error, we failed to do it.

For the average consumer of polling, this looks like two huge mistakes in back-to-back presidential elections. Are you worried about the reputation of polling, especially in an environment where the president has politicized the polls?

Sure. How could you not be if you take this at all seriously?

Polling has had its bad times. The irony is that we had generally been doing better at election outcomes before 2016. You have to earn that trust back, and the only way you do that is to have the next election do better. And in 2018, by and large, we did do better.

But then you have an error in ’20 that undoes that credibility-building of 2018. I insist that the Marquette poll did better than most of the polls in the state. Relatively speaking, we did quite well, and we did improve over the prior performance.

But I don’t really expect the public in Wisconsin or elsewhere to take away a nuanced view from this. The obvious and not wrong impression is that the polling had a really bad year. That means that for the next four years at least, we’ll be talking just like we are today about what went wrong with polling.

I don’t have a happy ending to this story except, you know, wait for 2024 and let’s hope we do a lot better.

Drop us a line!

We want to hear from our readers. Have a question? We’ll try to answer it. Have a comment? We’re all ears. Email us at onpolitics@nytimes.com.

… Seriously

“We are them. They are us. We, too, will disappear. We will become abstractions to be puzzled over by future people. That certainty, in the flux of 2020, feels anchoring. We are not unique. We move in the historical flow. The current moment will melt away like snow crust on a mustache.”

Read Sam Anderson on the delights of a very old snowball fight.

Thanks for reading. On Politics is your guide to the political news cycle, delivering clarity from the chaos.

On Politics is also available as a newsletter. Sign up here to get it delivered to your inbox.

Is there anything you think we’re missing? Anything you want to see more of? We’d love to hear from you. Email us at onpolitics@nytimes.com.