When rising British casualties in Afghanistan and Iraq started to raise public doubts 15 years ago, a new mantra began to be heard: Iraq was a war of choice, Afghanistan a war of necessity. The argument was that the US and its faithful ally, Britain, had launched an invasion in Iraq that was unjustified as it was based on a false premise: the hollow claim that Saddam Hussein possessed weapons of mass destruction.

The intervention in Afghanistan was different, it was said, even by many who opposed the Iraq war. Al-Qaida had organised the atrocities of 9/11 and its leader, Osama bin Laden, was based on Afghan soil. George W Bush was right to give the Taliban an ultimatum to hand him over or face invasion.

But here, too, there was a false premise, or indeed several. Mullah Omar and the Taliban leadership were as surprised to see the twin towers crash to the ground in New York as everyone else. They had never been consulted by Bin Laden on his strategy, let alone his targets. Anticipating US reprisals, Bin Laden and his large entourage of Arab fighters left Kandahar and hid in the Tora Bora mountains. Bush’s call on the Taliban to arrest him was unrealistic. So going after the Taliban was just as unnecessary as bringing regime change in Iraq.

It was also equally dubious from the standpoint of international law. There was no UN security council resolution authorising the US assault on Afghanistan. It was clear that Bush would want to punish al-Qaida for 9/11, but international law does not permit armed force for revenge or retaliation. The US claimed that al-Qaida had declared war on the US and it was entitled to respond with force in self-defence. International law only allows this if an enemy attack is imminent. In the autumn of 2001, imminence was hardly a relevant concept. None of the 19 9/11 hijackers was Afghan and they had mainly trained in Germany and the US. It had taken two years to prepare the attack, so there was no way al-Qaida could have mounted another similar atrocity imminently.

After 9/11, a few analysts argued that if the US was determined to use force it should have limited it to a search-and-destroy operation against al-Qaida in Tora Bora. Their view was ignored and Bush added a new war aim: the building of a modern democracy in Afghanistan. Joe Biden repudiated that in his speech on Monday when he stressed that US policy should be based on security from terrorism rather than any humanitarian reforms. His remarks are sparking a furious debate, but they are correct.

Britain, too, needs to re-examine its Afghan policies. It should hold an inquiry along the same lines as the Chilcot report into Iraq (except that it should report much faster). The first item on its agenda must be whether the decision to go for regime change in 2001 was wise or foolish. The events of the past two decades, culminating in the triumphant return of the Taliban that we have just witnessed, flows from that decision.

It is true that Kabul and other major Afghan cities have enjoyed 20 years of patchy progress. Women in particular have benefited and a generation of young people has grown up with the expectation of secure and free life choices. If the Taliban had not been ousted from power in 2001, none of this would have happened. But the country would have been spared the ravages and killing of the civil war that resumed in 2003 once the Taliban recovered from the shock of defeat. Like the Ashraf Ghani administration, it also just gave up in 2001 under the weight of US bombing with barely a shot fired. It was bound to seek ways to reverse it, however long it took.

In the century since Afghanistan gained independence from Britain in 1919, the country’s tragedy has been the constantly repeated cycle of defeat for the minority of Afghan modernisers who have sought to break the hold of conservative rural patriarchy. It happened with the first post-independence leader, Amanullah Khan, who took power on a wave of popularity but lost it after he introduced co-educational schools and stopped women wearing hijab, let alone the full burqa. Conservatives marched on Kabul in 1929, the army deserted and Amanullah abdicated.

Resistance to a new wave of reform arose again in the 1980s when Afghanistan’s communists, the People’s Democratic Party of Afghanistan (PDPA), expanded education for girls and increased opportunities for women to work outside the home. When they took Soviet support, they opened the door for an alliance of religious and tribal leaders (helped by western governments at the height of the cold war) to rise up as mujahideen warriors and brand the PDPA as atheists and lackeys of the Kremlin. When Moscow withdrew its aid in 1992 (like Trump and Biden today), the modernising regime quickly fell. Now we are seeing a third turn of the wheel of conservatives ousting reformers.

Observers wonder how the Taliban managed to achieve so sweeping a victory. The sad fact is that its patriarchal views are popular in rural and small-town Afghanistan and it could never have made its stunning military advances without local support. People had also lost faith in a corrupt central government and an army that the Pentagon was well aware was ineffective and unmotivated – as revealed in the “Afghanistan Papers”, hundreds of confidential interviews with US military and diplomatic leaders obtained by the Washington Post.

Many Afghans felt the Taliban produced quicker and more honest justice in village disputes between families. The UK government should have known this. Surveys commissioned for the Department for International Development in Helmand in 2010 showed that people preferred Taliban courts to the Kabul-appointed ones, where they had to bribe prosecutors and judges.

Afghans do not like invaders, whatever their motives, and the Taliban were able to exploit the narrative of patriotic resistance. Why did Britain ignore the lessons of history and follow the unhappy experience of the Soviet invasion and occupation? That must be the central issue in the public inquiry we need.

• Jonathan Steele is a former Guardian correspondent and author of Ghosts of Afghanistan: The Haunted Battleground