American Assn for Public Opinion Research

Experiments that yielded promising results in 2022 may not be enough if Trump is on the ballot again.With a highly successful polling cycle behind them, some pollsters believe a tactic that gained widespread adoption in 2022 may help carry them through the next presidential election. But even the tactic’s adherents say it may not be a panacea, particularly if former President Donald J. Trump is once again on the ballot.Pollsters have increasingly been weighting surveys based on whom respondents recall voting for in a previous election, in addition to adjusting for standard demographics such as race and age. This tactic has long been used in other countries to improve poll accuracy, but has become widely used in the United States only in recent years.“We are all terrified,” said Cameron McPhee, the chief methodologist at SSRS, CNN’s polling partner and a pollster that weighted some of its polls on recalled vote in 2022. She added, “We all feel good about the changes we made in 2022, but I think there is still a big question mark” headed into 2024.By weighting on recalled vote, pollsters can more easily correct partisan imbalances in who responds to polls, and in recent years Democrats have tended to respond to polls at higher rates than Republicans. Perhaps more important, weighting on recalled vote can specifically increase the influence of Trump supporters, a group that polls struggled to measure accurately in 2016 and 2020.The tactic’s adoption by pollsters in the United States remains far from universal. Several prominent pollsters achieved accurate results without it — including The New York Times/Siena College, which was named America’s most accurate political pollster by FiveThirtyEight after the 2022 cycle.Overall, 2022 was one of the most accurate years for polling in recent history, according to an analysis by FiveThirtyEight. Many pollsters “probably would have gotten 2022 right even without that extra weighting step, because we did,” said Patrick Murray of the Monmouth University Poll.After 2016, post-election analyses found that polls had consistently underrepresented less educated voters, who tended to disproportionately support Mr. Trump. To fix this, pollsters widely adopted education as an additional survey weight, and a cycle of accurate polls in 2018 seemed to reflect a return to normalcy.But in 2020, polls were more biased than they had been in any modern election, over-representing Democratic support by nearly five percentage points, as opposed to three percentage points — a more normal amount of error — in 2016.“I think one of the reasons 2022 was successful — and even to some extent 2018 — was that Trump himself was not on the ballot.” Mr. Murray said. “If history is any guide, we are probably going to see that nonresponse going into 2024 based on how the Republican nomination is going.”The 2020 election presented another distinct challenge — it took place amid the pandemic. Pollsters found that some Americans, stuck at home and lonely, were more likely to respond to surveys. While that was initially seen as a boon, it might have led to even more bias if it meant the uneven adherence to stay-at-home orders added another source of bias to who picked up the phone.Weighting on recalled vote is not without its concerns.Voters have been shown to have poor recall of whom they voted for or even whether they voted at all, typically being more likely to recall voting for the winner. One study of Canadian voters found up to a quarter of voters were inconsistent when recalling whom they had voted for.This misrepresentation of past vote can push polls in different directions depending on who won the most recent election. In 2022, that meant respondents were more likely to say they had supported Joe Biden, and pollsters using recalled vote would end up giving them less weight, meaning Republican support was bolstered.But with a prior winner from a different party, the effect would be reversed. An assessment by The Times found that weighting its 2020 polls using recalled 2016 vote would have made them even more biased toward Mr. Biden. And a report from the American Association for Public Opinion Research examining how 2020 polls could have been improved found that polls that weighted on recalled vote were no better than those that didn’t.Similarly, in 2022, weighting by recalled vote would have made Times/Siena polls less accurate. As published, without weighting on recalled vote, the final polls of Senate, governor and House races had an average error of less than two percentage points and zero bias toward Democrats or Republicans. When weighted using recalled vote to 2020 election results, average error would have increased by a percentage point, and overall the polls would have been slightly biased toward Republicans.But that might have been a consequence of other decisions The Times makes, which includes weighting to demographic information available on the voter file that is not always available to other pollsters.Other pollsters have found the recalled vote method to yield significant improvements over typical weighting schemes. SSRS used a range of weighting methods in 2022, including recalled vote for some of its polls, and also experimented with weighting on political identification. Its post-election analysis found that using recalled vote as a weight would have been the most accurate overall approach, increasing average accuracy by more than three percentage points over just weighting on standard demographics.“It’s a brute force method,” said Clifford Young, the president of U.S. public affairs at Ipsos. “That is, we don’t really know what it corrects for. Does it correct for only nonignorable nonresponse? Or does it correct for coverage bias? Or maybe a likely voter problem? Maybe all three.”Even so, pollsters are generally optimistic. “What the evidence is showing is that it gets us in a much better place in our polls than not using it,” Mr. Young said, noting that he believed most pollsters would be weighting on past vote in 2024. “I think the evidence thus far suggests it does more good than harm.” He added, “If we use the same weighting and correction methods that we used in 2020 in 2024, we’re going to miss the mark.” More