When Beto O’Rourke travelled to Yosemite in California to unveil his $5tn plan on climate change, a ripple of surprise crossed America. How did the tall white guy with the funny first name known for his punk past, Beatnik road trips and fondness for campaigning atop counters get to be the first Democratic candidate to proclaim on the crisis of our age?

This wasn’t the O’Rourke that the country had grown used to during his battle with Ted Cruz last November for a US Senate seat. Then, the Texas Democrat had propelled himself to within three percentage points of victory, and with it national stardom, by making viral speeches about NFL players taking a knee and by instilling hope through a feel-good but rather wishy-washy call to unity.

Now here he was framed against the beauty of Yosemite Falls, delivering a granular plan of action worthy of the most nerdish policy wonk. Coming from a politician from oil-rich Texas who has been criticized for his track record on fossil fuels, his proposals for the largest 10-year investment in history and a goal of net-zero emissions by 2050 caught many off guard.

“We were pleasantly surprised,” said David Turnbull of the climate advocacy group Oil Change US. “When you see someone like Beto O’Rourke calling for the elimination of fossil fuel subsidies and an end to fossil fuel leasing on public lands – that’s moving in the right direction.”

There was another group of people hoping to be pleasantly surprised by the Yosemite announcement that day – O’Rourke himself and his team of campaign advisers. They have been wrestling with one of the great magical mysteries of the early phase of the 2020 presidential election.

That is: the astonishing disappearing act of Beto O’Rourke.

Like Houdini, O’Rourke has gone from front of stage to a puff of smoke in six short months. #Betomania morphed into #Betofatigue, seemingly overnight.

Look back on the events of 7 November 2018, when he delivered his concession speech, having lost to Cruz in a packed sports stadium in El Paso, and you can see the contrast. At that time he was lauded as the politician who could do the impossible: challenge a virulent Republican like Ted Cruz in a solid red state like Texas and come within an inch of victory.

Next stop Donald Trump? But from the moment he launched his presidential bid in March, he has been struggling. Those very qualities that had been the recipe of his relative success in Texas suddenly became liabilities.

His charming ways and good looks were thrown back in his face as white privilege. That wasn’t helped when he gave Vanity Fair a gift of a one-liner on the eve of launch – “Man, I’m just born to be in it” – that made many Democrats wince.

The mere decision to run for the White House was interpreted as chutzpah. As the Daily Beast cruelly put it: “Reacting to losing to Ted Cruz by running for president is like failing to land a role in a community theater production and deciding to take your talents to Broadway.”

In the latest poll from Quinnipiac university, O’Rourke is drawing a glum 5% of Democrats and Democratic-leaning voters. He is being outgunned on 10% by Pete Buttigieg, the mayor of South Bend, Indiana, who has stolen much of his thunder.

“We’ve seen Mayor Pete take the lead in the newcomer department,” said Quinnipiac’s Peter Brown who predicted worse to come. “We’ve got 18 months to go and I bet there will be other fresh faces taking the spotlight.”

So what happens next to O’Rourke now that the spotlight has swung away from him? Can he complete the Houdini trick and make a reappearance? And if he can, what kind of potential president would he present to the American people?

‘He was always very focused’

Examining those questions, it quickly becomes clear that all roads Beto lead to El Paso. That’s the dusty, sunbaked border town in Texas where he was born Robert Francis O’Rourke in 1972.

His father, Pat, was a businessman and judge, and his mother, Melissa, ran a furniture store. They were comfortably off and formed part of the white middle class elite in a city that is 80% Latino.

O’Rourke’s opponents have tried to depict his youth as one of fecklessness and debauchery. Rightwing pundits like to poke him for the name “Beto”, claiming it is a conceit designed to suggest that he has Latino roots, which he does not.

They also point to a drunk-driving episode in 1998, his teenaged flirtation with his punk band Foss and to the period when he floundered around in New York City working as a glorified maid. Reuters recently contributed to that pile of potential negative attack material with the revelation that O’Rourke had secretly belonged to the prominent “hactivist” group Cult of the Dead Cow.

But those who have known O’Rourke for years say they do not recognize this caricature of the spoilt wild boy from the border town. Take Maggie Asfahani, a writer and El Paso restaurateur, who had a teenaged romance with O’Rourke when he was at an all-male boarding school in Virginia.

Asfahani clearly recalls their first encounter in an El Paso mall when he was back on holiday. Her memory instantly puts to rest any suggestion that “Beto” was an adult affectation. “I’d imagined this Mexican kid, given the name, but there was this really tall white guy. I can categorically dismiss all that speculation – he was ‘Beto’ at least since I’ve known him in high school.”

Asfahani can also, incidentally, put to rest any scurrilous talk about a much reproduced photograph of O’Rourke flanked by his Foss bandmates in which he wears a long floral dress.

“I want to put on the record, that is my dress he’s wearing,” she said. “There’s nothing particularly complicated about it – we were all hanging out, and someone thought it would be funny if we switched clothes, the girls and guys. That was all, just being different.”

What struck Asfahani then as now was something that’s been lost amid the presidential chatter – his seriousness. “He was always very focused. He was this fiercely intelligent, curious person who was into things, always wanting to learn things, always with a book in his hand.”

Asfahani remains in touch with O’Rourke to this day. She thinks the flak he has taken over unearned entitlement since he entered the 2020 race, based on her knowledge of the man, has been unfair.

“It strikes me he is finding his way on the national stage,” she said. “He’s being open and honest and vulnerable, hoping people will relate to that and see themselves in it. That’s not a fault: it has been his personality since I’ve known him.”

‘He learned how to take energy from crowds’

O’Rourke’s entry into politics followed his return to El Paso, the prodigal son, at age 26. Having been largely away since his teens, he re-engaged with the city, setting up Stanton Street, an internet company combined with a short-lived alternative newspaper.

His political ideas formed around his ambitions for El Paso, which in the late 90s was economically depressed and suffering from a brain drain of young people. O’Rourke forged a bond with four friends who came to be known as the Progressives, one of whom, Veronica Escobar, now occupies the El Paso congressional seat vacated by O’Rourke.

“What motivated him was the idea that El Paso didn’t have to settle for being a low-key, down-at-heel city which was fine with exporting its children,” said Bob Moore, former editor of El Paso Times who has known O’Rourke since his return in 1998.

The Progressives’ aspirations for their city led all four friends to stand for local office. All four won, with O’Rourke joining the El Paso city council in 2005.

Moore recalls that in his political infancy O’Rourke cut a paradoxically diffident figure for a man now competing for the White House. “By nature he’s a deeply private person. He was very awkward when he first ran for office, uncomfortable in large groups. Then he learned how to take energy from crowds, and that has changed him.”

Despite such initial reticence, O’Rourke championed some radical and highly contentious causes. He became a passionate advocate of legalization of marijuana long before it was de rigueur, authoring a book with fellow Progressive Susie Byrd, Dealing Death And Drugs, that argued powerfully that the US war on drugs was a disaster for both sides of the US-Mexican border.

He also fought to extend health benefits to unmarried and same-sex partners of city workers, then a hot potato in heavily Catholic El Paso.

You will hear O’Rourke projecting his track record on marijuana and LGBT rights on the presidential campaign trail. You are much less likely to catch any reference to a third controversy that dogged him as city councilor, and still does to this day: the redevelopment of downtown El Paso.

The plan to revitalize downtown with a new sports arena, Walmart and other facilities preceded O’Rourke’s time on the council, having been initiated in 2004. But he embraced it keenly.

His involvement became problematic for two main reasons. The first was his family ties to the mastermind behind the plan, multi-millionaire real estate magnate William Sanders. Months after O’Rourke joined the council, he married Amy Sanders and William Sanders became his father-in-law.

The downtown project was a private-public partnership. The private side involved a civic organization called the Paso del Norte Group, PDNG, which Sanders set up with some of his super-wealthy friends from El Paso.

Controversy erupted when it emerged that O’Rourke was also a member. Did his position, with one foot in the private PDNG side of the deal and another on the public council side, amount to a conflict of interest? He was slapped with an ethics complaint, later dismissed.

O’Rourke initially voted in the council to go ahead with the development plan, but as local resistance grew he recused himself from several key votes. Further cries of foul play descended on him in 2012, when O’Rourke made an insurgent’s bid to unseat the incumbent Congressman for El Paso, Silvestre Reyes.

A company owned by Sanders contributed $40,000 to a Republican-backed Super Pac that invested in attack ads against Reyes, contributing to O’Rourke’s underdog victory and giving him a leg-up to Washington.

In a recent interview with the American Prospect, O’Rourke denied any conflict relating to his father-in-law. Sanders “made it a rule that he religiously followed, never to talk politics”, he said.

But the Sanders connection still rankles with activists opposed to the downtown scheme such as David Romo, a leading member of the main protest group Paso del Sur. He said that O’Rourke’s connections to Sanders takes the shine off his current claim that as a presidential candidate he eschews big money and is running a “people’s campaign”.

Romo told the Guardian that in his view O’Rourke’s role in the redevelopment casts doubt on his 2020 candidacy. “What happened in El Paso tells me that the solution to our national problems does not come from a multi-millionaire funded by billionaires who does their bidding.”

Romo is a celebrated historian of El Paso’s revolutionary past and as such is an articulate exponent of the second criticism leveled at O’Rourke over the redevelopment scheme – that he sided with gentrification despite the harm it would inflict on poor Latino residents and historic El Paso. “He was the pretty face of ugly gentrification.”

O’Rourke denies that he sided with gentrifiers, insisting his intention was to breathe new life into the dilapidated heart of a major city. He did tell the American Prospect, though, that in hindsight he accepts that he did “a really poor job of listening to that criticism”.

‘He really does need to answer questions’

Similar controversy followed O’Rourke to Washington. Whether it originated from his innate pragmatism as a politician who tends to decide each issue as it comes rather than following ideology, or whether it was because of his roots in Texas, a state that has been dominated by Republicans for the past 20 years, his voting record in Congress was striking for its lack of party purity.

Although El Paso veers overwhelmingly Democratic, a fivethirtyeight.com tracker shows that he voted 30% of the time in line with Trump. Compare that to his presidential rivals: Kamala Harris (17%), Bernie Sanders (14%) or Elizabeth Warren (13%).

That didn’t matter much in his senatorial race last November. But then he was running against Ted Cruz, one of the most toxic rightwing senators who even fellow Republicans call “Lucifer in the flesh”.

In that race he proved himself to have several of the qualities that might appeal to Democratic voters looking for a presidential nominee capable of beating Trump, first and foremost his ability to turn out the vote. He showed himself adept in appealing to young people, African Americans, Latinos and suburban white women – electoral groups all likely to play a crucial role in 2020 in deciding Trump’s fate.

But the road to the presidential nomination is proving to be a stonier path for O’Rourke than his route last year. By taking his campaign national he has moved on to much more fertile ground for a Democrat than the traditionally arid soil of Texas, yet it has come at the price of sharply intensified scrutiny.

Which brings O’Rourke back to his climate change announcement amid the splendor of Yosemite Falls. Fossil fuel activists may have been pleasantly surprised by O’Rourke’s robust policy, but that doesn’t mean they have forgotten that his relationship with the oil industry has been complicated.

He hesitated for weeks before agreeing to sign the No Fossil Fuel Money Pledge in which candidates forego all donations above $200 from Pacs, lobbyists and executives of fossil fuel companies. The pledge was particularly sensitive for O’Rourke, who according to Open Secrets accepted more contributions from oil and gas in 2018 than any congressional candidate other than Ted Cruz.

He has said his hesitancy was out of concern for ordinary workers in the industry who should be allowed to participate. The organizers of the pledge however stressed that only the donations of top bosses were excluded.

In the end, he did sign the pledge, two days after his Yosemite declaration.

Another sticking point is that O’Rourke voted twice in Congress to lift a 40-year ban on US exports of crude oil. He tried to justify the vote in October 2015, two months before the Paris Agreement on combating climate change was adopted by 195 nations, by arguing that US crude was cleaner than that of other countries and “the oil that supplies the current dominant mode of transportation will have to come from somewhere”.

The lifting of the ban has led to a massive spike in US crude exports, from well under 1m barrels per day to more than 3m per day currently. “There’s been a dangerous and problematic increase in the extraction of crude oil driven by exports in the US. He really does need to answer questions about that vote,” David Turnbull of Oil Change US said.

It all points to the steep uphill climb that Beto O’Rourke faces if he is to claw his way back into the Democratic spotlight. The Yosemite announcement made a solid start, introducing American voters to a more serious, focused politician than they had previously been shown.

Now the real scramble begins.