Why Fox’s Call on Arizona, Which Was Right, Was Still Wrong

If you’re a subscriber to this newsletter, my guess is you’d be interested in my colleague Peter Baker’s article about the drama at Fox News in the aftermath of its decision to call Arizona for Joe Biden on election night.

Here’s the short version: Fox News executives, news anchors and pundits were enraged over the call, with messages and a recording showing they thought it hurt ratings and threatened to “impact the brand” by alienating Donald J. Trump’s supporters.

Most people would agree that political and branding concerns shouldn’t dictate an election call by a news organization. But the article has nonetheless rekindled an old debate about whether Fox News was really “right” to call Arizona for Mr. Biden on election night in 2020.

This debate can be a little confusing, since Fox was right in the most important sense: It said Mr. Biden would win Arizona, and he ultimately did.

But a race call is not an ordinary prediction. It’s not like calling heads or tails in a coin toss. A race call means that a candidate has something like a 99.9 percent chance of winning. As a result, a call can be wrong, even if the expected outcome ends up happening. If you assert that there’s a 99.9 percent chance that a coin flip will come up heads, you’re wrong — regardless of what happens next.

Of course, everyone knows heads or tails is a 50-50 proposition. It’s much harder to know whether Mr. Biden had a 50.1 percent or 99.9 percent chance of winning Arizona based on the data available at 11:20 p.m. Eastern on election night, when Fox called the state for Mr. Biden. Most other news organizations didn’t think so; only The Associated Press, a few hours later, joined Fox in making the call so quickly. And in the end, Mr. Biden won Arizona by just three-tenths of a percentage point — a margin evoking a coin flip.

Was the Fox call the result of the most sophisticated and accurate modeling, or more like being “right” when calling heads in a coin flip? It appears to be the latter — a lucky and dangerous guess — based on a review of televised statements by the Fox News decision team and publicly available data about the network’s modeling.

The Fox team believed Mr. Biden would win Arizona by a comfortable margin at the time the call was made, based on erroneous assumptions and flawed polling. While it worked out for Fox in the end, similarly risky decisions could have easily led to a missed call, with potentially dire consequences for trust in American elections.

I should disclose that I’m not an entirely disinterested party. Here at The Times, we rejected the A.P. call on Arizona (The Times usually accepts A.P. calls, but we independently evaluate A.P. projections in very important races) because we couldn’t rule out a Trump victory based on the available data. I believe we were right about that decision. But much as the Fox team has an incentive to argue its case, readers may believe that I have an incentive to argue against the Arizona call. I should also disclose that I know and like the Fox News decision desk director Arnon Mishkin.

In a recording of a Fox Zoom meeting two weeks after the election obtained by The Times, Mr. Mishkin acknowledged that the Arizona call appeared “premature” but that “it did land correctly.”

A Fox spokesperson on Sunday said that “Fox News continues to stand by its decision desk’s accurate call of Arizona.”

Still, there is a compelling body of publicly available evidence suggesting that Fox, when it called the state, fundamentally misunderstood the remaining votes. It did not imagine that Mr. Trump could come so close to winning.

Why Fox made the call

At the time Fox called Arizona, Mr. Biden led Mr. Trump by 8.5 percentage points, with an estimated 73 percent of the expected vote counted. The tabulated votes were mainly mail ballots received well ahead of the election. To win, Mr. Trump needed to take about 61 percent of the remaining votes.

In addition to the tabulated vote, the Fox decision desk also had the Fox News Voter Analysis, otherwise known as the A.P. VoteCast data — a pre-election survey fielded by The Associated Press and NORC at the University of Chicago. The AP/NORC data showed Mr. Biden ahead by six percentage points in Arizona.

A person with knowledge of how the call was made, who spoke on condition of anonymity, said that the Fox team believed that the early returns confirmed the Fox News Voter Analysis. Indeed, Mr. Biden’s early lead seemed to match the survey’s findings among early voters, who broke for Mr. Biden by 10 points in the survey, 54 percent to 44 percent. The implication was that Mr. Biden was on track for a clear victory.

When asked on election night on Fox to explain the Arizona call, Mr. Mishkin rejected the notion that Mr. Trump would do well in the outstanding ballots. Instead, he said he expected Mr. Biden to win the remaining vote:

“We’ve heard from the White House that they need to get just 61 percent of the expected vote and they’ll be getting that.” He added: “But the reality is that’s just not true. They’re likely to only get 44 percent of the outstanding vote.” 

These figures were repeated by Daron Shaw, a Republican pollster on the Fox decision desk, and Mr. Mishkin in subsequent appearances. At the various times these statements were made, Mr. Biden would have been on track to win the state by between four and nine percentage points if the outstanding vote had gone so heavily in his favor.

Through a Fox News spokesperson, Mr. Mishkin said he “misspoke on election night” when he said Fox expected Mr. Biden to win the remaining vote. If Mr. Mishkin did misspeak, there was still no indication that the Fox team expected Mr. Trump to win the remaining votes by a meaningful margin — let alone an overwhelming margin.

On air on election night, Mr. Mishkin offered two main reasons to expect Mr. Biden to fare well in the remaining vote:

“Yes, there are some outstanding votes in Arizona. Most of them are coming from Maricopa, where Biden is currently in a very strong position. And many of them are mail-in votes, where we know from our Fox News Voter Analysis that Biden has an advantage.”

On their face, these arguments weren’t outlandish. Mr. Biden won Maricopa County, which is the home of Phoenix and a majority of Arizona voters. He won the mail vote in Arizona as well.

In the end, Mr. Trump won 59 percent of the remaining vote, all but erasing Mr. Biden’s advantage.

What Fox missed

How could a group of mostly mail-in and mostly Maricopa ballots break for Mr. Trump by such a wide margin? 

The reason was foreseeable before election night.

While “mail” votes sound monolithic, there can be important differences between mail ballots counted before and after the election. That’s because Arizona counts mail ballots in roughly the order in which they are received, and different kinds of voters return their ballots at different times.

Ahead of the election, it was clear that Democrats were turning in their ballots earlier than Republicans. As a result, the mail ballots counted on election night — those received at least a few days before the election — were likely to break for Mr. Biden by a wide margin.

The flip side: The voters who received mail ballots but had not yet returned them were very Republican. If they ultimately returned their ballots, these so-called “late” mail ballots counted after the election would break heavily for Mr. Trump.

It wasn’t inevitable, of course, that Mr. Trump would win these ballots by as wide a margin as he ultimately did. It was possible that many of these Republicans would simply vote on Election Day. In the midterms last November, for instance, Republicans failed to decisively win the “late” mail vote under fairly similar circumstances.

But in 2020, whether the late ballots would be overwhelmingly Republican was nonetheless “the big question,” as I wrote before the election. As a result, we never contemplated the possibility of a call in Arizona on election night; it was an easy decision for us to reject the A.P. call without knowing exactly how the “late” mail ballots would break.

When asked on television the day after the election if the so-called late mail voters could back Mr. Trump with more than 60 percent support, Mr. Mishkin dismissed the possibility, saying it could happen “if a frog had wings.”

Mr. Mishkin said he did not “ascribe any significance” to whether mail voters turned in their ballots on Election Day. Instead, he expected the “late” ballots would “confirm” their call. He was confident the late data “would look like the data we’ve noticed throughout the count in Arizona,” which to that point had shown Mr. Biden with a clear lead.

Similarly, Mr. Shaw said in a radio interview the day after the election that “we don’t have any evidence” that “late” early voters would break for Mr. Trump.

In fairness to Fox News and The A.P., it was hard to anticipate the difference between early and late mail ballots ahead of the election. It required marrying a detailed understanding of absentee ballot returns with an equally deep understanding of the mechanics of how Arizona counts mail ballots.

The Fox News Voter Analysis was a factor here again as well. The survey offered no indication that mail voters surveyed near the election were likelier to back Mr. Trump, according to the person with knowledge of the call. And previously, late-arriving mail ballots in Arizona had benefited Democrats.

But the ballot return data showed that this time could be very different. In the end, it was.

Models and polls that missed the mark

Analytical and research failures are inevitable. No one can perfectly anticipate what will happen on election night, especially in the midst of a pandemic. What matters is whether these failures yield a bad projection, and here the quality of statistical modeling — and especially whether the model properly quantifies uncertainty — becomes an important factor.

Fox’s statistical modeling was highly confident about its Arizona call. On election night, Mr. Mishkin said, “We’re four standard deviations from being wrong” in Arizona. This implied that the Fox model gave Mr. Trump a 1-in-10,000 chance of victory.

It’s hard to evaluate why the model was so confident. What’s clear is that it provided a basis for Fox to call the race, even as there were mounting nonstatistical reasons to begin to doubt the estimates.

By the time of the Arizona call, it was already clear that the AP/NORC survey data — along with virtually all pre-election polling — had overestimated Mr. Biden. In North Carolina, for example, Mr. Trump had already taken the lead after AP/NORC data initially showed Mr. Biden ahead by five points. The same data initially showed Mr. Biden ahead by seven points in Florida, where Mr. Trump was by then the projected winner.

As a result, there was already reason to be cautious about estimates showing great strength for Mr. Biden. But rather than become a source of uncertainty, Mr. Biden’s positive numbers in the AP/NORC data appeared to become a source of confidence — as Mr. Biden’s strength in the early vote appeared to confirm expectations.

One indication that Fox’s modeling was prone to overestimate Mr. Biden was its publicly available probability dials, which displayed the likelihood that Mr. Biden or Mr. Trump would win the key battleground states.

At various points, these estimates gave Mr. Biden at least an 87 percent chance of winning Ohio and at least a 76 percent chance of winning Iowa; Mr. Trump ultimately won both by nearly 10 points.

Maybe most tellingly, Fox gave Mr. Biden a 95 percent chance to win North Carolina — even at a point when it was quite obvious that Mr. Trump would win the state once the Election Day vote had been counted.

Through a Fox News spokesperson, Mr. Mishkin said, “The program that translated the decision desk’s numbers into the probability dials was not working properly at times.” Fox stopped using the probability dials on air, though they remained available online.

But even if the dials were erroneously overconfident or otherwise not exactly to Fox’s liking, they nonetheless erred in almost exactly the same way as the Arizona call. In all four states, including Arizona, the AP/NORC data greatly overestimated Mr. Biden; the early vote count leaned heavily toward Mr. Biden; and the Fox estimates confidently swung toward Mr. Biden.

Whether it was inaccurate AP/NORC data, misunderstanding the “late” mail vote, technical issues or overconfident modeling, there’s not much reason to believe that there was a factual basis for a projection in Arizona. It came very close to being wrong. If it had been, it could have been disastrous.

The public’s confidence in elections would have taken another big hit if Mr. Trump had ultimately taken the lead after a call in Mr. Biden’s favor. It would have fueled the Trump campaign’s argument that he could and would eventually overturn the overall result. After all, he would have already done so in Arizona.