Joe Biden exuded confidence. “We’re going to win,” the US president told reporters before departing Cop26 in Glasgow. “I think we’re going to win in Virginia.”

But as Biden returns to Washington, he faces questions about why his prediction was so wrong – and whether Democrats’ loss in the most important election of the year will send his presidency into a downward spiral.

The Republican Glenn Youngkin’s surprise victory over the Democrat Terry McAuliffe in the race for governor of Virginia is a brutal rebuke for Biden, who had personally invested in the race, twice making the short trip from Washington to campaign for McAuliffe at rallies.

It will particularly sting because Donald Trump, whom he defeated in Virginia by 10 percentage points in last year’s presidential election, will doubtless seek to claim credit for the result and savor his revenge.

But the truth is that this election was more about the current president than the spectre of the last one.

Biden’s ambitious agenda has stalled in Congress. By his own admission, the inertia has sucked oxygen away from priorities such as a police reform and voting rights, disillusioning the activists who fuel Democratic turnout. Inflation and gasoline prices are up. Global supply chains are buckling. And Biden’s sunny predictions for post-withdrawal Afghanistan were as off the mark as his predictions for Virginia.

The president’s sagging approval rating of 42% combined with historical headwinds to drag McAuliffe down. Nothing energizes a political movement like opposition: the president’s party has lost every election for governor of Virginia over almost half a century – the exception was McAuliffe himself in 2013.

But this time McAuliffe failed to inspire. The chairman of Hillary Clinton’s 2008 presidential primary campaign had a distinct whiff of Clinton 2016: a career politician imbued with a sense of entitlement who constantly found himself on the defensive against an upstart candidate drawing bigger crowds.

Like Hillary Clinton’s reference to “deplorables”, McAuliffe made a perceived gaffe – “I don’t think parents should be telling schools what they should teach” – which was replayed endlessly in Youngkin attack ads.

McAuliffe’s central argument – that Youngkin is an acolyte of Trump – was about the past. Youngkin’s central argument – that schools are under attack from culture warriors on race and gender – was about the future, even if it was riddled with falsehoods. To many voters, the future tends to be more persuasive.

Enough of them did not seem to know or care that Youngkin’s arguments on schools were based on a lie. He stoked fears about critical race theory being taught in schools – it isn’t – with a caricature of Black children learning to think they are victims and white children learning to self-hate.

It cut through and proved effective in a febrile, pandemic-era atmosphere where parents shout and even turn violent at school board meetings debating issues such as gender identity and mask mandates. Whereas McAuliffe wanted to nationalize the election, Youngkin managed to keep it local, albeit by tapping into Fox News talking points following last year’s Black Lives Matter protests.

Expect this incendiary mix of children and racism to be chapter one of the Republican playbook in next year’s midterm elections for Congress. Expect chapter two to be How to Deal with a Problem Called Donald Trump.

The 45th president will still be welcome in the safe districts of the Make America Great Again nation, sure to draw fanatical crowds and turn out the vote. But in swing states, Youngkin has shown Republicans the way to have their cake and eat it too.

In the Republican primary, he praised Trump and fanned his false claims of voter fraud by raising concerns about “election integrity”. In the general election, he was willing to tacitly pat Trump on the back without ever embracing him – he eschewed mentions of the former president in campaign speeches and must have been tremendously relieved that Trump never turned up in person.

Youngkin squared the circle that many Republicans have struggled with, creating a template for how to win over moderates and independents without alienating the Trump base, or vice versa. Call it the Goldilocks principle of strategic ambiguity: neither too hot nor too cold, but just the right temperature.

Democrats knew exactly what he was doing. McAuliffe relentlessly tried to conflate Youngkin with Trump. At a rally last week, Biden warned: “Extremism can come in many forms. It can come in the rage of a mob driven to assault the Capitol. It can come in a smile and a fleece vest. Either way, the big lie is still a big lie.”

But it was all in vain.

Youngkin, like Trump, might have emphasized his status as a businessman and political outsider but otherwise came over as a suburban dad, more polished and less profane: the acceptable face of Trumpism. Yet his tactics were just as dark, dishonest and divisive.

Democrats will now need to find a counter-strategy fast. Some commentators have suggested that members of the House and Senate could desert Biden and rush to the exits, retiring rather than facing a bloodbath in the midterms, so weakening the president’s hand at a crucial moment for his agenda. Virginia is a warning cry that the party needs strong leadership to get it done before things fall apart.

Wednesday marks the first anniversary of Biden’s defeat of Trump in a presidential election like no other. But the pandemic of Trumpism rages in new and unexpected ways – and the Youngkin variant may prove among the most dangerous.