A Key Polling Warning Sign Has Dimmed Since September

Way back in September, we noticed a warning sign in the polls: Democrats were showing strength in exactly the places where the polls overestimated their chances in 2020.

The pattern raised the possibility that solid Democratic leads in several key Senate races were a mirage — the result of the same biases that led the polls to overestimate Democrats in those same states two years earlier.

With the election only days away, that warning sign is gone: There is no longer any material relationship between relative Democratic or Republican strength in the key Senate races and the polling error from 2020.

It’s worth clarifying a common misperception about our earlier article: The “warning sign” wasn’t simply Democratic strength, it was where the Democrats were strong. Equivalent Democratic strength in a different set of states — say, a huge Democratic lead in Nevada but a deficit in Wisconsin — would not have been a warning sign. But a much stronger set of results for Republicans across the board would still have counted as a warning sign.

Similarly, what matters now from a polling perspective isn’t that Democrats are faring worse in the surveys. It’s where the Democrats show relative strength and weakness, and whether that fits the pattern from the 2020 election. Now, only Ohio stands out as a place where the Democrats are both faring unusually well and where the polls erred last time. Conversely, only Nevada counts as a relative weakness. Elsewhere, there’s not much pattern at all.

If anything, the pattern now comes closer to resembling the polling error of 2018, not 2020. The polls still struggled in important ways in the 2018 midterms, but on balance they were not as biased toward Democrats as they were in 2016 or 2020.

The absence of a relationship between Democratic strength in the polling averages and the 2020 polling error doesn’t prove there won’t be another 2020-like polling error this time around. Indeed, there are still signs that many traditional pollsters are plagued by nonresponse bias, including our Times/Siena poll.

But two years ago, the similarity between Joe Biden’s strength and the 2016 polling error did turn out to be a harbinger of the polling misfire to come.

What has changed? The polling in several critical states has come into better alignment with the FiveThirtyEight fundamentals — the benchmark we’re using to determine whether Democrats are faring better or worse.

There are at least two components to the change, each with different consequences for the likelihood of a polling misfire.

First: There has been movement toward the fundamentals and away from the 2020 polling error pattern.

Here, the same pollsters tell a very different story than they did over the summer, with Republicans gaining in the places where they showed relative strength and Democrats holding up or gaining in places where the Republicans had shown strength.

Take Wisconsin’s Senate race as an archetypal example: The same polls that showed the Democrat Mandela Barnes faring surprisingly well over the summer now show Ron Johnson in the lead. While it’s possible that nonresponse bias faded or that pollsters made some adjustments, it seems likelier that the race simply moved toward Mr. Johnson as the primary season ended and as Republicans attacked Mr. Barnes.

A Republican polling lead in a state like Wisconsin should make pollsters sleep better at night. In 2016, there wasn’t really any polling that showed Mr. Johnson leading in Wisconsin, even though he went on to win by around 3.5 points. And two years ago, virtually every poll showed Mr. Biden up big in Wisconsin. Perhaps Mr. Johnson will win by, say, 12 points instead of three or four points, as the polls are showing today. But the fact that Mr. Johnson’s lead lines up with expectations makes such a large miss seem a lot less likely.

Not every shift toward the fundamentals, however, so clearly seems to reduce the risk of polling error. Imagine, for instance, if the polls moved for a reason that might merit a change in our understanding of the fundamentals.

Here, Pennsylvania would be an excellent example. On paper, the fundamentals suggest a close race; in theory, any shift toward Dr. Mehmet Oz ought to reduce the risk of a large polling error. But John Fetterman’s health is an unusual case. This isn’t just a natural return to the fundamentals; it’s a distinct development that arguably should change our expectations. It might be more comparable to the effect of a scandal — the kind of event that might lead us to doubt whether the fundamentals apply. If Mr. Fetterman’s health status means we ought to move our expectations from a close race toward a comfortable Oz victory, the potential for polling error might remain just as large.

Second: Some of the movement in polling averages is because of changes in the composition of the pollsters.

While there has definitely been some movement toward Republicans since the summer, in many cases Republican gains have been supercharged by an entirely different factor: a flood of low-cost, usually Republican-backed surveys in key battleground states. These polls tend not to adhere to industry standards for transparency or data collection. (I went into this in more depth in Saturday’s newsletter.) And not surprisingly, they have tended to be much more favorable for Republican candidates than the media or university-sponsored surveys that have typically dominated the polling averages in years past — including as recently as this summer.

As a result, there were a lot of states where the polling averages moved to the right, simply because more right-leaning pollsters joined the fray.

It’s helpful to go back to our Pennsylvania example. Mr. Fetterman and Dr. Oz are tied in our average, but most of the traditional surveys from university or media outlets, once considered the gold standard, show Mr. Fetterman with a lead — even if those pollsters do show him in a tighter race than they did over the summer. Indeed, there isn’t a single traditional type of survey with Dr. Oz ahead. Instead, the average is tied because a list of firms with a Republican-lean show Dr. Oz ahead.

There’s no need to weigh in on whether the Republican-leaning or traditional firms are “right,” at least not in this article. While the cheaper firms often have deeply and obviously unrepresentative samples — like showing Republicans with 30 percent support among the Black vote, as opposed to the typical 10 percent or less — the nonpartisan polls might still struggle with the same kinds of nonresponse bias that plagued the industry in 2020.

Indeed, the response rate among white registered Democrats was 28 percent higher than among white registered Republicans in the most recent wave of Times/Siena Senate polls. This is about the same as it was in 2020.

What matters for the topic at hand isn’t which group of pollsters is right or wrong. What matters is that the flood of Republican polls has helped to dim or even turn off the warning sign in the polling averages. Even if the underlying data remains as skewed as it was over the summer or in 2020, the emergence of these Republican-leaning pollsters has reduced the risk that polling averages will overestimate the Democrats.