Donald Trump was caught with classified documents and Democrats were outraged. Joe Biden was caught with classified documents and Republicans were outraged. Mike Pence was caught with classified documents and it became clear that there might be a bigger problem here.

America has a crisis of “overclassification”, critics say. Since the 11 September 2001 terrorist attacks, Washington has been overzealous in defining government secrets. Politicians and officials can too easily fall foul of this secrecy-industrial complex but the biggest losers are the American people denied democratic accountability.

Among the prominent voices calling for reform is Jameel Jaffer, executive director of the Knight First Amendment Institute at Columbia University in New York. Previously at the American Civil Liberties Union (ACLU), he fought court battles over landmark post-9/11 cases relating to national security and individual rights.

Jaffer makes no excuses for former president Trump, who hoarded about 300 documents with classified markings at his Mar-a-Lago estate in Florida and resisted justice department efforts to retrieve them. He regards the Biden and Pence cases as different because, as far as is known, they inadvertently left classified material at their respective homes in Delaware and Indiana and willingly turned it over to authorities.

Jaffer would have expected the former vice-presidents to be more careful but argues that there is a more fundamental point: the failure of a process in which the government classifies about 50m documents every year – at a cost to taxpayers of approximately $18bn – while not declassifying them at anything like the same rate.

“The bigger scandal here is not any particular episode involving the mishandling of classified information but rather the classification system itself, which is totally broken in ways that are bad not just for national security but for democracy,” Jaffer, 51, said this week by phone from Brooklyn, New York.

“There’s too much information that’s classified. Too many people have access to the classified secrets. A lot of the information is classified for the wrong reasons because its disclosure would embarrass somebody or it would be inconvenient or would subject government officials to scrutiny that they would rather not have.”

Special counsel investigations into Trump and Biden are just the tip of the iceberg.

This week the National Archives wrote to representatives of living former presidents and vice-presidents requesting that they check their personal papers in case classified documents are still among them. Former officials from all levels of government discover they are in possession of classified material and turn them over to the authorities at least several times a year, the Associated Press reported.

Why all the secrecy? One explanation is incentives. Classification can be useful for a government official seeking to conceal incompetence, preserve a bureaucratic monopoly on a particular set of facts or keep a rival government agency in the dark. Conversely there is no penalty for keeping information – however trivial or unnecessary – secret and no mechanism for declassifying in the public interest.

One consequence of this runaway effect is that the national security bureaucracy suffers classification overload: when everything is secret, nothing is secret. Jaffer commented: “That has national security implications because it means that it’s harder to keep track of and protect the secrets that really do need to be secret.

“It also breeds a kind of cynicism because people see, on the one hand, senior government officials going on about how sensitive these secrets are and, on the other hand, treating the documents in this kind of careless way.”

There is a double standard, he added, between the way senior officials and junior employees are treated when they mishandle classified material. “That, too, is bad for national security because it demoralises intelligence community employees.”

Rapacious classification also takes a toll on democracy. “A lot of the information that the public needs is unjustifiably kept out of the public domain and, as a result, public debate about important issues like foreign policy and war and counter-terrorism policy is impoverished or, even worse, distorted by needless secrecy.”

Jaffer discovered this firsthand at the ACLU, which he had joined as a volunteer to advocate for people detained in raids in immigrant communities around New York in the weeks after September 11. Over the next 14 years he worked on cases relating to CIA black sites, the interrogation and torture of prisoners, indefinite detention, the drone campaign and warrantless wiretapping.

He added: “The government made bad decisions in secret and, by the time the public learned of those decisions, it was too late to avoid some of the costs.”

September 11 was a turning point after decades in which classification principally related to discreet wars overseas or the development of weapons, including nuclear weapons. The reaction to the attacks on New York and Washington changed the character of government secrets and brought them much closer to home.

Jaffer commented: “After 9/11, a lot of this had much more direct implications for individual rights including the constitutional rights of Americans. There’s a difference between what is the government doing in south-east Asia and what is it doing here in New York City.

“There’s a difference between keeping secret the specifications for a particular weapon and keeping secret the fact that you’re torturing prisoners in overseas black sites or engaged in dragnet surveillance of Americans’ phone calls and emails. Those are different kinds of secrets: they go to government policy, the scope of government power, the meaning of individual rights. The public has a much stronger interest in an informed public debate about those kinds of questions.”

If the system is broken, what can be done to fix it? Presidents Bill Clinton and Barack Obama sought to encourage declassification with limited success. Jaffer would like to see an institution outside the executive branch – perhaps the judiciary – given the authority to make national security information public where the public interest outweighs the need for secrecy.

“One foundational flaw in our national security system is that public interest balancing never happens. There is nobody who is tasked with considering the possibility that the government might have some interest in keeping something secret but the public interest in disclosure is greater.

“There’s no public interest balancing in the context of the Freedom of Information Act. If you sue for national security information and the government says the information is classified, that’s the end of it. The judges don’t then say, well does it really need to be classified? But they should be empowered to do that. That would be an important reform.”

The Espionage Act of 1917 is also long overdue a rewrite, according to Jaffer.

In the 20th century only one person, Samuel Loring Morison, was convicted under the act for sharing information with the press (he was pardoned by Clinton in 2001). But after September 11, both Democratic and Republican administrations have used it aggressively to target journalists’ sources including Reality Winner, Terry Albury and Chelsea Manning.

More recently the government has invoked the Espionage Act to go after a publisher: Julian Assange, the founder of WikiLeaks, whose methods Jaffer likens to those of journalists reporting on national issues. “They communicate confidentially with their sources, protect their sources’ identities, solicit classified information, publish government secrets.

“Those are the things that Assange is being prosecuted for and that national security journalists engage in all the time – and have to engage in order to do the work we want them to do. That’s why I see the Assange case as such a threat to press freedom.”

Jaffer is not an absolutist who wants to put all information in the public domain. But nor does he accept that the leaking of government secrets is an existential threat.

“The much bigger problem is not that sensitive things are being disclosed dangerously but rather that important information crucial to the public’s ability to understand government policy, and crucial to the democratic legitimacy of the government’s policies, is being withheld unjustifiably,” he said.

“What we need is a bottom-up reform of the entire classification system including the Espionage Act. I don’t think this is a system that is serving us well. The fact that the system is so broken has very significant costs for our society, and it’s bad not just for public debate and for democracy but even for national security too.”