Joe Biden suffered a bitter political blow early on Wednesday when Democrats went down in a shock defeat in the election for governor of Virginia.

The Democratic candidate, Terry McAuliffe, had campaigned with Biden and Barack Obama but it was not enough to prevent the Republican Glenn Youngkin pulling off an upset.

The AP called the race for Youngkin in the early hours of Wednesday morning. The Republican took an early lead after polls closed that he maintained throughout the evening, while McAuliffe lagged in key counties that Biden swept in 2020.

Clenching his fists then clapping his hands, Youngkin addressed jubilant supporters in Chantilly just after 1am. “All righty, Virginia, we won this thing!” he exclaimed. “How much fun!”

The 54-year-old political neophyte described it as “a defining moment” for millions of Virginians “sharing dreams and hopes”. Youngkin promised: “Together, we will change the trajectory of this commonwealth and friends, we are going to start that transformation on day one. There is no time to waste.”

In a nod to what became his defining campaign issue, the Republican said: “We are going to restore excellence in our schools… We are going to introduce choice in our public school system… Friends, we’re going to embrace our parents, not ignore them. We’re gonna press forward with a curriculum that includes listening to parents’ input.”

As on the campaign trail, Youngkin did not utter the name “Trump”.

The battle in Virginia has been seen as a litmus test of Biden’s presidency one year after he won the White House, and it coincided with his agenda stalling in Congress and his approval rating sinking to 42%.

“The fight continues,” said McAuliffe in a speech on Tuesday night, thanking his campaign staff for a hard-fought race, but stopping short of a concession.

“We’ve got to make sure we protect women’s right to choose here in the commonwealth of Virginia. We’ve got to make sure everyone gets quality, affordable healthcare here in the commonwealth of Virginia. Everybody’s entitled to a world-class education here in the commonwealth of Virginia and we are going to continue that fight tonight, and every day going forward.”

McAuliffe’s all-out effort to portray Youngkin as an acolyte of Donald Trump proved less effective than the Republican’s laser-like focus on whipping up parents’ fear and anger about culture war issues in Virginia’s schools.

Youngkin made false claims that critical race theory – an analytic framework through which academics examine the ways that racial disparities are reproduced by the law – is rampant in the state’s education system (in fact it is not taught).

His campaign zeroed in on a perceived gaffe by McAuliffe at one of their debates: “I don’t think parents should be telling schools what they should teach.”

Importantly, Youngkin was also successful walking a political tightrope in which he accepted Trump’s endorsement but never mentioned him in stump speeches or invited him to campaign with him in person. He cultivated sufficient ambiguity to appeal to moderate Republicans without alienating the Trump base.

History was on Youngkin’s side in that the party that loses the White House tends to be energised and usually wins the Virginia’s governor’s race a year later. But McAuliffe himself had bucked that rule when he became governor in 2014 (he was limited to one term).

However, no Republican had won statewide office since 2009, and Biden beat Trump in Virginia by 10 percentage points, meaning that a Democratic loss here would reverberate across the nation.

Meanwhile, in New Jersey, a similar story was unfolding as the Democratic governor, Phil Murphy, fought to win re-election against his Republican challenger, Jack Ciattarelli. That race was too close to call, with Ciattarelli narrowly ahead. If Murphy holds his office, he would be the first Democrat re-elected as the state’s governor in 44 years, while a defeat would bode ill for national Democrats.

At McAuliffe’s election night event at a hotel in Tysons, Virginia, giant TVs that had been showing cable news coverage were switched off long before any result was finalized. Stunned supporters trailed out into the chilly night.

Manisha Singh, 48, an analyst, said: “It’s extremely disappointing. I can’t imagine all the lies that were spread to influence voters. I was knocking on doors and people were saying take the porn out of public schools, which is a lie. They were just repeating what they had seen on Fox News.”

Singh added: “This might be a huge wake-up call to Democrats. We always play nice when the other party spreads lies. We need to be more aggressive. “

Argument quickly broke out among national Democrats over what had gone wrong. The Progressive Change Campaign Committee said in a statement: “Terry McAuliffe sadly can blame his loss on a few corporate-aligned obstructionist Democrats who blocked bold action in Congress, plus his own reliance on backward-looking Trump messaging.

It added: “Democrats won’t win simply by branding one opponent after another as a Trump clone, and then hoping to squeak out a razor-thin win. When Democrats fail to run on big ideas or fulfill bold campaign promises, we depress our base while allowing Republicans to use culture wars to hide their real agenda.”

McAuliffe, a career politician and establishment Democrat, is likely to point to Biden’s falling popularity and Washington gridlock as factors in his defeat. Youngkin, a former executive at the private-equity firm the Carlyle Group, sold himself as a political outsider challenging the liberal elite.

His strategy – a delicate balancing act of stoking culture wars in education and winking at Trump without fully embracing him – is seen as a potential blueprint for Republican candidates in next year’s congressional elections.

Trump said in a statement: “All McAuliffe did was talk Trump, Trump, Trump and he lost! What does that tell you, Fake News? I guess people running for office as Democrats won’t be doing that too much longer. I didn’t even have to go rally for Youngkin, because McAuliffe did it for me.”

Dan Conston, president of the Congressional Leadership Fund, a super political action committee dedicated to electing Republicans to the House of Representatives, said: “Tonight’s results put every Democrat in Congress on notice.

“Virginia was once ground zero for suburban decline but has now become the epicenter of a Republican comeback in 2022. If Republicans can win even in a state so blue that Joe Biden won by 10 points, then far more Democrats are in peril next year than they want to admit.”

The battles in Virginia and New Jersey came as voters in states across the US headed to the polls on Tuesday, an off-year election day, to cast their ballots for local governors, mayors and public measures.

In New York, the former police officer Eric Adams easily won his race to become the next mayor of New York City. In Boston, Michelle Wu made history when she defeated Annissa Essaibi-George to become the first woman of color and Asian American elected as the city’s mayor. History was also made in Durham, North Carolina, where Elaine O’Neal became the city’s first Black female mayor after campaigning on neighborhood safety, housing and economic relief in the aftermath of the pandemic, while Abdullah Hammoud won the mayoral race in Dearborn, Michigan, making him the city’s first Arab American leader.

In Minneapolis, voters rejected an initiative that would have replaced the police department with a new department of public safety, nearly a year and a half after the police killing of George Floyd inspired nationwide protests against police brutality.

Maanvi Singh and the Associated Press contributed reporting