In the summer of 2009, the latest in a long line of US military commanders in Afghanistan commissioned the latest in a long line of strategic reviews, in the perennial hope it would make enough of a difference to allow the Americans to go home.

There was some excitement in Washington about the author, Gen Stanley McChrystal, a special forces soldier who cultivated the image of a warrior-monk while hunting down insurgents in Iraq.

Hired by Barack Obama, McChrystal produced a 66-page rethink of the Afghan campaign, calling for a “properly resourced” counter-insurgency with a lot more money and troops.

It quickly became clear there were two significant problems. Al-Qaida, the original justification for the Afghan invasion, was not even mentioned in McChrystal’s first draft. And the US could not agree with its Nato allies on whether to call it a war or a peacekeeping or training mission, an issue with important legal implications.

In the second draft, al-Qaida was included and the conflict was hazily defined as “not a war in the conventional sense”. But no amount of editing could disguise the fact that after eight years of bloody struggle, the US and its allies were unclear on what they were doing and who they were fighting.

The story is one of many gobsmacking anecdotes and tragic absurdities uncovered by Craig Whitlock, an investigative reporter at the Washington Post. His book is based on documents obtained through freedom of information requests, most from “lessons learned” interviews conducted by the Special Inspector General for Afghanistan Reconstruction (Sigar), a watchdog mandated by Congress to keep tabs on the hundreds of billions flowing into Afghanistan.

In the Sigar files, and other interviews carried out by military institutes and research centres, Whitlock found that soldiers of all ranks and their civilian counterparts were “more open about their experiences than they likely would have been with a journalist working on a news story”.

Blunt appraisals were left unvarnished because they were never intended for publication. The contrast with the upbeat version of events presented to the public at the same time, often by the very same people, is breathtaking.

The Afghanistan Papers is a book about failure and about lying about failure, and about how that led to yet worse failures, and so on for 20 years. The title and the contents echo the Pentagon Papers, the leaked inside story of the Vietnam war in which the long road to defeat was paved with brittle happy talk.

“With their complicit silence, military and political leaders avoided accountability and dodged reappraisals that could have changed the outcome or shortened the conflict,” Whitlock writes. “Instead, they chose to bury their mistakes and let the war drift.”

As Whitlock vividly demonstrates, the lack of clarity, the deception, ignorance and hubris were baked in from the beginning. When he went to war in Afghanistan in October 2001, George Bush promised a carefully defined mission. In fact, at the time the first bombs were being dropped, guidance from the Pentagon was hazy.

It was unclear, for example, whether the Taliban were to be ousted or punished.

“We received some general guidance like, ‘Hey, we want to go fight the Taliban and al-Qaida in Afghanistan,’” a special forces operations planner recalled. Regime change was only decided to be a war aim nine days after the shooting started.

The US was also hazy about whom they were fighting, which Whitlock calls “a fundamental blunder from which it would never recover”.

Most importantly, the invaders lumped the Taliban in with al-Qaida, despite the fact the former was a homegrown group with largely local preoccupations while the latter was primarily an Arab network with global ambitions.

That perception, combined with unexpectedly easy victories in the first months, led Bush’s defense secretary, Donald Rumsfeld, to believe the Taliban could be ignored. Despite offers from some leaders that they were ready to negotiate a surrender, they were excluded from talks in December 2001 on the country’s future. It was a decision the United Nations envoy, Lakhdar Brahimi, called the “original sin” of the war.

Rumsfeld declared there was no point negotiating.

“The only thing you can do is to bomb them and try to kill them,” he said in March 2002. “And that’s what we did, and it worked. They’re gone.”

Not even Rumsfeld believed that. In one of his famous “snowflake” memos, at about the same time, he wrote: “I am getting concerned that it is drifting.”

In a subsequent snowflake, two years after the war started, he admitted: “I have no visibility into who the bad guys are.”’

The Taliban had not disappeared, though much of the leadership had retreated to Pakistan. The fighters had gone home, if necessary to await the next fighting season. Their harsh brand of Islam had grown in remote, impoverished villages, honed by the brutalities of Soviet occupation and civil war. The Taliban did not represent anything like a majority of Afghans, but as their resilience and eventual victory have shown, they are an indelible part of Afghanistan.

Whitlock’s book is rooted in a database most journalists and historians could only dream of, but it is far more than the sum of its sources. You never feel the weight of the underlying documents because they are so deftly handed. Whitlock uses them as raw material to weave anecdotes into a compelling narrative.

He does not tell the full story of the Afghan war. He does not claim to do so. That has to be told primarily by Afghans, who lived through the realities submerged by official narratives, at the receiving end of each new strategy and initiative.

This is a definitive version of the war seen through American eyes, told by Americans unaware their words would appear in public. It is a cautionary tale of how a war can go on for years, long after it stops making any kind of sense.