The statistics are familiar but remain startling: America’s incarceration rate per 100,000 is “roughly twice that of Russia’s and Iran’s, four times that of Mexico’s, five times of England’s, six times Canada’s” and nine times that of Germany. In addition, “parole and probation regulate the lives of 4.5 million Americans” – more than twice as many as are confined in prison.

These numbers come at the beginning of Bill Keller’s smart, short new book, in which he tries to explain how America became so addicted to mass incarceration, and how we might finally reform a system which houses a disproportionally Black and brown population.

Keller is a veteran journalist who won a Pulitzer for his first New York Times posting as a foreign correspondent, in Moscow as the Soviet Union collapsed. He went on to be executive editor and then a columnist, but in 30 years, criminal justice was never one of his specialties. That all changed when Neil Barsky, a journalist turned investor turned philanthropist, tapped Keller to be founding editor of The Marshall Project, an ambitious effort to produce great journalism about the “causes and consequences” of mass incarceration.

Keller’s book highlights many of the best pieces by Marshall Project reporters, but he also uses plenty of his own reporting to illuminate this particularly dark side of American democracy.

The “good news”: the incarcerated population has actually been in slow and steady decline, from a peak of 2.3 million in 2008 to 1.8 million in 2020, including an unprecedented drop of 14% spurred by early releases because of Covid.

America’s unfortunate exceptionalism on this subject is actually a fairly recent development. From the 1920s through the 1970s, the rate of incarceration mostly held steady at around 110 out of every 100,000 Americans. But it is nearly 500 today.

Liberals and conservatives were equally responsible. A Democratic House speaker, Thomas “Tip” O’Neill, sharply overreacted to the crack cocaine overdose of Len Bias, a Boston Celtics draftee, pushing through the 1986 Anti-Drug Abuse Act, “which imposed mandatory sentences, asset forfeitures and outlandishly severe sanctions on crack cocaine” favored by Black ghetto residents, while white consumers of powdered cocaine faced much more lenient penalties.

As Keller writes, “Rehabilitation was denigrated on the right as coddling”. But a Democratic Senate judiciary committee chairman, Joseph R Biden of Delaware, made everything much worse by championing the Violent Crime Control and Law Enforcement Act of 1994, which not only spurred a prison-building boom but also eliminated Pell Grants for prisoners enrolled in college courses. President Biden has acknowledged his mistake.

It was President Reagan who inserted the profit motive into the prison business, allowing the Corrections Corporation of America to pioneer “the idea of privately run, for-profit prisons”. As Keller explains, “Since the new prison owners were paid the same way as hotel proprietors, by occupancy, they had no incentive to prepare prisoners for release.” Private prisons now house about 7% of state inmates and 17% of federal.

Keller makes an unintentional argument for sending more Republicans to jail, by pointing out that three of the more unlikely advocates of prison reform are Republican officials who ended up in prison.

Patrick Nolan was the minority leader of the California assembly when, in 1993, he was indicted on charges of racketeering and extortion. He served 25 months in a federal prison near San Francisco. When he was paroled, he was recruited by Charles Colson, a famous Watergate felon from Nixon’s White House who found religion “shortly before serving seven months himself in a federal prison”.

Colson campaigned for more humane treatment of prisoners. Nolan became director of a new Center for Criminal Justice Reform at the American Conservative Union Foundation. Meanwhile, Bernard Kerik, Rudy Giuliani’s police commissioner who then did three years in federal prison for tax fraud and other crimes, became an advocate for voting rights for ex-felons.

It’s not all good news. By the end of Trump administration, Nolan had succumbed to a rightwing conspiracy theory that “billionaire George Soros was masterminding a ‘Trojan horse’ strategy to elect soft-on-crime prosecutors and bring down the entire criminal justice system”.

Keller points to Norway and Germany as providing the best examples for systemic reform. While American prison guards rarely get more than a few weeks of training, Germans get two years of college courses in psychology, ethics and communication. American visitors to German jails are amazed to see unarmed guards “shooting baskets, playing chess, sharing lunch” and having conversations with prisoners.

One reason Europe is so far ahead is its depoliticization of the criminal justice system: judges and district attorneys are appointed, not elected.

A Fordham University professor, John Pfaff, has pointed out that in the US, during the 1990s and 2000s, “as violent crime and arrests for violent crime both declined, the number of felony cases in state courts” suddenly shot up. Because of political pressures, “tens of thousands more prosecutors” were hired, “even after the rising crime of the 1980s had stalled out”.

Pfaff attributed the racial inequality in numbers of prisoners to “an imbalance of political power – tough-on-crime prosecutors elected by suburban whites who see the community destruction of mass incarceration from a distance”.

Keller reports the most effective ways to reduce the prison population are also the most obvious ones:

• Make low-level drug crimes “non-crimes”.

• Divert people into “mental health and addiction programs, or probation or community service”.

• “Abolish mandatory minimum sentences and encourage” judges to “apply the least severe punishment appropriate under the circumstances”.

• Give “compassionate release to old and infirm inmates” who don’t pose a real threat to the general population.

The challenge is to get these common-sense ideas to prevail over the rhetoric of politicians who still rail against anyone who is “soft on crime” – the knee-jerk ideology which got us into this catastrophe in the first place.

• What’s Prison For? Punishment and Rehabilitation in the Age of Mass Incarceration is published in the US by Columbia Global Reports