Bob Shrum had known Bill Clinton since their student days at Georgetown University in Washington. Working as a political consultant, he was at the White House to advise on the US president’s State of the Union address when it was rocked by one of the scandals of the century.

“I was walking up the driveway and camera crews were trying to chase me to get me to comment on it,” Shrum recalls of the first reports that Clinton had an affair with Monica Lewinsky, a 22-year-old intern. “The president, when we all assembled in the cabinet room, was very down and just said, well, we’ll reconvene tomorrow.

“That’s when I went into the Oval Office and said what I had to say … He’s very big and put me up against the wall and said, ‘I’ve known you a long time. I just want you to know I did not do this thing.’ I wasn’t going to make a judgment about whether I thought he did or he didn’t. I suspected he had.”

Indeed he had. Clinton initially lied about the relationship before apologising. Now the 1998 saga that nearly brought down a president – he was impeached by the House of Representatives but acquitted by the Senate – is about to play out all over again for a new generation in the splashy TV drama Impeachment: American Crime Story, starting on Tuesday.

Previous American Crime Story series have centred on the criminal trial and acquittal of OJ Simpson over the death of his wife Nicole Brown, and the assassination of the fashion designer Gianni Versace. The new chapter stars Beanie Feldstein as Lewinsky, British actor Clive Owen as Clinton and Edie Falco as his wife Hillary.

It promises to finally put Lewinsky’s experience at the centre of a narrative usually told by tabloid newshounds and political historians. She served as a producer and script consultant on the show and formed a close bond with Feldstein.

Lewinsky told the Hollywood Reporter: “The first time that there was all this attention on this story, in 1998, I wasn’t prepared in any way. And now there’s going to be a lot of attention on this story again, but I’m in a very different place in my life. I just turned 48, I’ve now been a public person for half my fucking life. So … we’ll see.”

Clinton had been elected in 1992 at 46, making him the youngest president since John F Kennedy and inviting comparisons for charm and charisma. His reckless affair with intern Lewinsky began on 15 November 1995 after they fell into conversation and she admitted: “I have a huge crush on you.”

The following April she was transferred to the Pentagon where she met Linda Tripp (played by Sarah Paulson in Impeachment: American Crime Story), a disillusioned civil servant who had also come from the White House and who bitterly resented the Clintons.

Lewinsky confided in Tripp about her affair with the president. In the autumn of 1997, Tripp contacted literary agent Lucianne Goldberg with a view to exposing the scandal in a book. Now 86 and living in New Jersey, Goldberg recalls: “I said, do you have pictures, do you have letters, do you have anything that is going to make a publisher say, ‘Wow, this is really good stuff and she’s proved it’, because it’s a pretty racy story.

“She said I don’t know how to put something down that would absolutely prove what I’m saying. I said, well, how do you know this woman? She said she works next door to me at the Pentagon and we talk every day and she’s having this affair with the president and she tells me every gruesome detail of the thing. So I said bingo!, here’s what you do: you get in a cab, you go down to RadioShack, you tell them you want a tape recorder that works on your telephone and you tape the woman and that way you’ve got a story. It was a pure business deal.”

Did she or Tripp have any moral qualms about secretly recording Lewinsky without her knowledge? “Oh, I don’t think morality came up on it. She had a story to tell and I was in the business of selling stories. What would I regret? I never didn’t tell the truth and I never took a penny, so my conscience was clear and, frankly, I was in business. I don’t make morality judgments on business unless it’s, ‘Do you want to sell a baby?’ or something.”

The plan worked. On one of the tapes, Lewinsky can be heard saying: “Linda, I don’t know why I have these feelings for him. I never expected to feel this way about him. And the first time I ever looked into his eyes close up and was with him alone, I saw someone totally different than I had expected to see. And that’s the person I fell in love with.”

Meanwhile an investigation was long under way into whether the Clintons had been involved in fraudulent land deals in their home state of Arkansas. The Whitewater inquiry, as it was known, was led by a zealous independent counsel named Kenneth Starr, a former official in the George HW Bush administration.

The trail went cold and Starr faced pressure to close the case. But when Tripp disclosed the existence of the Lewinsky tapes, Starr sensed an opportunity to take the hunt in an entirely new direction. He sent Tripp to meet Lewinsky at the Ritz-Carlton Hotel, where FBI agents seized Lewinsky, took her to an upstairs room and used threats to pressure her to turn against the president. Sobbing uncontrollably, she refused.

A day later, however, Clinton was testifying under oath in a sexual harassment lawsuit brought by Paula Jones, a former Arkansas state employee. If the president lied about his relationship with Lewinsky, Starr could charge him with perjury. In the event, Jones’s lawyer used a such a specific legal definition of sex that Clinton tried to split hairs by claiming he never had “sexual relations” or “an affair” with Lewinsky.

Nevertheless, exposure of the relationship was now inevitable. Tripp’s book never materialised but Goldberg alerted her old friend Matt Drudge, an online gossip columnist, and soon the news of an illicit affair between the American president and a White House intern sent shockwaves around the world. Clinton’s aides and cabinet members were stunned and did not know what to believe.

Dick Morris, a political consultant, told American Experience on the PBS channel: “When the Lewinsky scandal broke the president paged me and I returned the call. And he said, ‘Ever since I got here to the White House I’ve had to shut my body down sexually I mean, but I screwed up with this girl. I didn’t do what they said I did, but I may have done so much that I can’t prove my innocence.’

“And I said to him, ‘The problem that presidents have is not the sin, it’s the cover-up and you should explore just telling the American people the truth.’ He said, ‘Really, do you think I could do that?’ And I said, ‘Let me test it, let me run a poll.’ So I took a poll and I tested popular attitudes on that and I called him back and I said, ‘They will forgive the adultery, but they won’t easily forgive that you lied.’”

But Clinton did not follow the advice and continued to deny a relationship with Lewinsky. “I did not have sexual relations with that woman, Ms Lewinsky,” he protested indignantly. “I never told anybody to lie. Not a single time. Never. These allegations are false, and I need to go back to work for the American people.”

He repeated the lie to Hillary, who then asserted that there had been a “vast rightwing conspiracy” against her husband ever since he ran for president. The notion of an attempted coup would become the heart of the Clinton team’s counterattack against Starr, whose credibility took a further hit last year when he defended Trump at a Senate trial following his impeachment for improperly seeking help from Ukraine to boost his chances of reelection.

So, was Clinton the victim of a witch-hunt?

Paul Rosenzweig, who was senior counsel to Starr during the investigation, says: “My opinion on that has changed over time. At the time, I did not credit it at all. I was of the view then – and still am now – that presidents who lie under oath have committed crimes and those need to be fully considered.

“At the time, I was also convinced that was the motivating factor for everybody in our office and, slight tinges of partisanship aside, we were all devoted to the principle of the rule of law and the idea that no person, no president was above the law.

“My retrospective view of what was happening at the time with respect to Clinton for others in the Starr organisation has to take into account that both Ken Starr and his successor, Bob Ray, were on President Trump’s team defending him at the first impeachment trial. I cannot square that defence of Trump with the importance of the rule of law that we ascribed to our activities under Clinton.”

Clinton’s denials were shattered when Lewinsky agreed to testify in exchange for immunity from prosecution. Under the terms of the deal, she submitted a blue dress stained with Clinton’s semen. The president was forced to confess all to his wife, answer questions before Starr’s grand jury and eat humble pie before the nation: “Indeed I did have a relationship with Ms Lewinsky that was not appropriate. In fact it was wrong.”

But he resisted demands for his resignation and, in September, Starr delivered the results of his investigation to Congress. Over 475 pages, the Starr report contained very little about Whitewater but plenty of prurient and salacious detail. It accused Clinton of perjury, obstruction of justice and abuse of office.

The effort backfired. The public seemed to have concluded that Clinton’s personal conduct, while painful for his family, did not interfere with his official duties and should not be used as a political weapon. Today, in the wake of the #MeToo movement, the judgment on a powerful man abusing his position would surely be very different and compel him to resign.

Bill Whalen, a research fellow at the Hoover Institution at Stanford University in Palo Alto, California, says: “I’ve got to believe he’s gone in the #MeToo era because this is about power. It’s about not just seducing and taking advantage of a young woman but then also the way they try to bury her with a defense department job and then try to send her to New York and try to silence her. It was power at the end of the day.”

In 1998, however, most voters seemed willing to forgive. Democrats actually gained seats in the midterm elections, prompting the resignation of Newt Gingrich as House speaker. But Republicans still held the majority and channeled their obsession into charges of obstruction of justice and perjury, making Clinton only the second president in history to be impeached.

Gingrich says: “The great problem for Clinton, which he and his allies never wanted to confront, was that he committed perjury. It wasn’t what he did with Lewinsky; it was the fact that as a lawyer who knew better under oath he lied about it, and perjury is a felony. I thought that Starr actually did the whole thing a disservice because he got fascinated with the sexual parts of it and that really wasn’t the heart of the issue.”

But earlier this year another Republican former House speaker, John Boehner, admitted that he regretted the impeachment because it was politically motivated. Even Gingrich now wishes it had been done “more carefully”.

He explains: “We needed to focus much more on the devastating impact of perjury and we needed to go after Clinton and his lawyers on why they engaged in perjury and the degree to which they themselves would have agreed that it was wrong. I also think that you could have ended up with some kind of censure, which would have required Clinton to do what he finally did, which is admit he was wrong.”

Clinton was found not guilty by the Senate and so remained in office. The damage to his legacy, however, was enduring. His vice-president, Al Gore, narrowly lost the 2000 election to George W Bush, who had run with a promise “to restore honour and dignity” to the White House. Clinton’s brand of centrist politics has fallen out of fashion in the Democratic party; he was relegated to a minor speaking slot at last year’s national convention.

Joe Conason, a journalist and co-author of The Hunting of the President, says: “Clinton has to consider it one of the greatest regrets of his life that he made this mistake and that it cast a shadow over him. The reason they impeached him was so that this would be the first line in the obituary, that this is inescapable now as a verdict on him and his presidency.”

But there is a deeper story about overreach by his rightwing enemies and misconduct by the independent counsel, Conason argues. “This was a political prosecution that was always intended to try to take down the Clinton administration through fair means or foul. That’s where Monica comes into the story, but it’s a story that began long before anybody heard of her. It’s important for people to understand that context.”

• Impeachment: American Crime Story starts on FX in the US on 7 September with a UK date to be announced