Fifty years ago on Sunday, Lyndon Baines Johnson died. He was 64, and had been out of power since stepping down as president in 1969, in the shadow of the Vietnam war. Forty-five years later, in 2018, the Guardian marked the anniversary of his death. The headline: “Why Lyndon Johnson, a truly awful man, is my political hero.”

Mark Lawrence laughs.

“I think I read that one,” he says.

It seems likely. Lawrence, a distinguished Vietnam scholar, is director of the LBJ Presidential Library in Austin, Texas.

Johnson was a Texas Democrat who rose through Congress to be vice-president to John F Kennedy, then assumed the presidency when Kennedy was killed. From 1963 to 1969, Johnson presided over great social reform at home and gathering disaster abroad. His legacy has never been less than complex, his place in American culture attracting historians by the hundred and big-name actors in droves. Bryan Cranston, Brian Cox and Woody Harrelson have recently played LBJ.

Lawrence continues: “One of the ideas that an awful lot of people hold about LBJ, and I think it’s not wholly incorrect, but it’s problematic, is that he was this vulgar, crude man who used four-letter words and demeaned his subordinates and threw temper tantrums.

“There’s no question that Caro” – Robert Caro’s biographical masterwork – “is the go-to source for the uglier parts of his personal style. But I think you can also make an argument, and Caro I think comes around to this view in the later books, that LBJ managed to combine whatever elements of that old caricature hold up with a genuine sense of compassion for ordinary people.

“Many biographers see the link between his own hardscrabble youth and the struggles of his family and a peculiar sensitivity to the plight of the downtrodden, which certainly affected his view of racial discrimination. The sensitivity to poverty, whether it affected Black, brown or white, came from his own experience.

“My writing about LBJ has largely been critical, but I don’t have any difficulty saying this was a man with a genuine sense of compassion.”

Lawrence is speaking to mark the half-century since the 36th president died. LBJ is in the news anyway. He was the architect of the Great Society, overseeing the passage of civil rights protections and a welfare system now under renewed attack. Joe Biden often compares his post-Covid presidency to that of Franklin D Roosevelt amid the Great Depression, but comparisons to Johnson are ready to hand.

Lawrence says: “The points of similarity are remarkable. The guy with long service in the Senate” – Johnson from Texas, 1949-1961, Biden from Delaware, 1973-2009 – “the guy who could cross the aisle, the guy who spoke in pragmatic, bipartisan terms. Both of these guys became vice-president to a younger, less experienced but much more charismatic person” – Biden to Barack Obama – “and that was kind of their ticket to the presidency.

“But I think some of these comparisons are ultimately unfair to Biden, because the political context is just so different. My own view is, sure, LBJ deserves credit for being this enormously persuasive, forceful guy who knew how to bend people to his will. But at the end of the day, LBJ was pushing an open door.

“Even the Civil Rights Act and the Voting Rights Act, these great achievements, they passed by big margins. There were bipartisan coalitions. LBJ deserves credit for his ability to put those coalitions together. But … I think it’s possible to exaggerate LBJ’s importance and to forget the importance of Hubert Humphrey, Jacob Javits and Everett Dirksen, all key players as well.”

Bipartisan players, too. Humphrey was LBJ’s vice-president, Javits a Republican senator from New York, Dirksen, of Illinois, Republican Senate minority leader.

“I think that’s precisely what’s lacking now. The situation is so polarised that you could bring LBJ back from the dead and he’d be an utter failure in this political context, because his skills would have been meaningless in the context of 2023.”

Biden passed a coronavirus rescue package, an infrastructure bill and the Inflation Reduction Act, all meant to help the US recover from Covid, with razor-thin margins in Congress and against Republicans gone to extremes. LBJ’s shadow may be long – at a shade over 6ft 3in he is the second-tallest president, after Abraham Lincoln – but Biden does not necessarily labour within it.

So how might progressives see Johnson? If they read Caro, they will learn how he came from a world of Texas populism, tinged with socialism, that now seems far gone indeed.

“At least by the standards of the era,” Lawrence sees in LBJ “a genuine willingness to think hard about poverty and how to insulate people against economic forces over which individuals had no control whatsoever.”

Whether teaching in a dirt-poor school in Cotulla in 1928 or working “for the National Youth Administration in the 1930s, LBJ shows glimmers of his willingness to cross racial lines and to think seriously about the situation of African Americans and Mexican Americans”.

To Lawrence, the Texas years “indicate that LBJ was an unusual person, for a southern white man who came of age in the 1920s and 30s.”

In the 1950s, when Johnson led the Senate, he defended white supremacy. As president, he oversaw the Vietnam disaster. But Charles Kaiser, a Guardian contributor and author of 1968 in America and The Gay Metropolis, also sees the bigger picture.

“In 1968, I hated Lyndon Johnson with all my heart, because I was 17 – and lived in fear of being drafted. Fifty years later, it is clear three other things about his presidency were much more important than the war that destroyed him.

“The Civil Rights Act of 1964 and the Voting Rights Act of 1965 made him the most courageous president since Lincoln. Johnson may or may not have said ‘We have lost the south for a generation’ after he signed the 1964 law, but he certainly knew that was true. By fighting for those two laws, he did more to redeem the promise of the Emancipation Proclamation than any president before him.

“Medicare is the third prong of a noble legacy. It did more to improve the lives of senior citizens than anything else except Franklin Roosevelt’s social security. A hundred years from now, I think Johnson will be considered one of our greatest presidents.”

To Lawrence, Johnson’s reputation is “mixed. But I think the mix of impressions is quite different from what it was certainly 30 years ago.

“When he died, and for many years thereafter, Vietnam hung so heavily over LBJ that he was a little bit radioactive … it was something conservatives and the left could agree on. Vietnam was a debacle and LBJ bore principal responsibility for it. But I think in the last decade and a half, there’s been a gradual reappraisal.

“The level of dysfunction and partisanship in Washington has led people to take another look at LBJ and how he was able to work across the aisle and achieve so much. There’s a kind of longing, I think, for that kind of political effectiveness.

“So many of the issues that LBJ worked on are back with us, and I think this has led at least parts of the political spectrum to have a new appreciation for him.

“In a period when immigration and the environment and voting rights are under threat in a profound way, people are rediscovering LBJ as someone who maybe didn’t have perfect answers but worked very effectively, at least by the standards of recent decades, and achieved real results.”