The US was poised to set the Taliban on the path to diplomatic recognition before the plan was derailed by the Afghan rulers’ sudden U-turn on a promise to allow girls’ education, the Guardian understands.

The group prompted international outrage and confusion on Wednesday when it reneged on a deal to allow teenage girls to go to secondary school, just a week after the education ministry announced that schools would open for all students.

US diplomats had been so optimistic that the Taliban would make good on the promise that a joint event had been planned ahead of this weekend’s Doha Forum in Qatar that would have set the process in motion to grant diplomatic recognition to the group.

A seat had been reserved for the Taliban at a panel at the forum dedicated to girls’ education in which a Taliban representative would have addressed the role of women with Afghan female activists.

The sudden reversal undermined the argument that a more “moderate” leadership now dominates the Taliban, and such optimism was further clouded this weekend when the group ordered Afghanistan television stations to remove BBC news bulletins in Pashto, Persian and Uzbek.

In a statement on Sunday the BBC said “This is a worrying development at a time of uncertainty and turbulence for the people of Afghanistan. More than 6 million Afghans consume the BBC’s independent and impartial journalism on TV every week

Western officials made it clear that diplomatic recognition will be impossible unless the decision on girls’ education is reversed. The move will also make it harder the international community to raise money for an international pledging conference next week, and require tighter handling of any cash raised so that it does not

Thomas West, the US special envoy for Afghanistan, said: “I was surprised by the turnaround this last Wednesday and the world has reacted to it by condemning this move. It is a breach first and foremost of the Afghan people’s trust.

“I believe hope is not all lost. I am hopeful we will see a reversal of that decision in the coming days.”

But West defended the US engagement with the Taliban saying that a complete diplomatic rupture would mean abandoning 40 million Afghans amid growing concerns over a possible famine in the country.

“We are talking about the modalities of an urgent humanitarian response, the need for more than a humanitarian response, a policy not just an admire the problem of a broken banking sector but find ways to fix it, a professionalisation of the Central Bank so that the international financial community can begin to have confidence in it, we are talking about terrorism and we are talking about women’s rights.

“One of the first times we sat down in October in a formal setting they had a request of us ‘please put our civil servants – 500,000 – back to work’. We thought a logical place to start given the sector resonated so much in the international community was education. We had requests of them, as well. Number one, women and girls could attend at all levels across large swathes of Afghanistan. Number two we wanted to see a monitoring mechanism and third there be a serious and rigorous curriculum. Over the following the months the international community receive the necessary assurances, and more importantly the Afghan people were told on March 23 we would see girls attend secondary education and that did not occur.”

Hosna Jalil, a former interior minister was one of many Afghan women at Doha to claim the Taliban will not be able to keep a lid on the demand for education. She said the last 20 years had not been a waste but left a positive legacy. “We facilitated a generation, two thirds of the population, that knows what a better life looks like. That is why we will not give up. They are loud, they believe in freedom and democracy.”

Malala Yousafzai, who won the 2014 Nobel Peace Prize for her fight for all children’s right to education, told the Forum that times had changed since the Taliban first banned girls’ education in 1996.

“It is much harder this time – that is because women have seen what it means to be educated, what it means to be empowered. This time is going to be much harder for the Taliban to maintain the ban on girls’ education. They are learning in the hide-outs. They are protesting on the streets. This ban will not last forever. They were waiting outside the school gates in their uniforms and they were crying. Seeking education is a duty of every Muslim,” she said.

Dalia Fahmhy, an Afghan professor of political science said in 1999 no girls were in secondary schools. “Within 15 years later there were 3.7 million girls. Over that period a thousand women became business owners. This cannot be curtailed. We live in a digital age and 68 % have cellphones and 22 % are connected to each other and to the world. This cannot be curtailed. 27 % of the parliament were women.”