The US presidency was supposed to be something different, something novel, compared with the fossilised monarchical rule that it supplanted after the American revolution. Born of Enlightenment theory, settler colonialism and 18th-century warfare, the US constitution gave the chief executive primarily an enforcement role, with the authority to lead armed forces in the event of foreign encroachment or domestic unrest but stripped of the capacity to legislate or issue judicial decisions. The architects of the new republic meant for the president to preside over a citizenry well-endowed with rights, not to rule over cowed subjects.

Chief executives from George Washington and Abraham Lincoln to Franklin Roosevelt and Ronald Reagan have been sorely tested by both the responsibilities and the limitations of the country’s highest, loneliest office. Through civil war, economic catastrophes, foreign misadventures, social upheavals and plagues, the presidency has endured, but it – and the 45 men who have occupied the job – has been moulded and often humbled by the promise and perils of the office.

Is the US presidency – indeed, American democracy – equal to the dire challenges of the 21st century? One could certainly argue that it isn’t, based on the ongoing bungling of the Covid-19 response, the horrifying (and presidentially inspired) insurrection of 6 January 2021 and the glacially slow and fickle efforts to address everything from climate change to widening social inequality. If the founding fathers meant to circumscribe the power of the presidency out of a well-founded fear of kingly abuses, then they would surely comprehend the creeping threat that authoritarianism and political extremism present to the US system of government today. Nevertheless, they probably could not have guessed that the hard lessons that they had learned about the fragility of democracy would be so fiercely resisted or blithely ignored more than two centuries after they beseeched a patrician general from the Virginia countryside to preside over their fledgling experiment in government by the people.

Of the many works that I have found useful in thinking about the history of the US presidency and for writing my newest book, The Black President: Hope and Fury in the Age of Obama, these 10 have been among the most helpful. They are a mix of biographies, memoirs and reportage which, taken together, represent some of the best writings by and about the small group of powerful people who have occupied the White House.

1. Never Caught: The Washingtons’ Relentless Pursuit of Their Runaway Slave, Ona Judge by Erica Armstrong Dunbar (2017)Dunbar’s important book is less a biography of George Washington, Martha Washington, or Ona Judge, the runaway enslaved woman whom the first couple made such extraordinary efforts to recapture, than a look into the power and privilege of a slaveholding elite forcing its way through a new republic rhetorically committed to liberty. The relentless pursuit of Judge by the Washingtons after her bold flight from the new US capital in Philadelphia is expertly told by Dunbar.

2. The Hemingses of Monticello: An American Family by Annette Gordon-Reed (2008)This history of overlapping, intertwined families vivifies the world around Thomas Jefferson, the third US president, while skilfully making more legible the travails and aspirations of the enslaved people on his storied estate at Monticello. The decades-long relationship between Jefferson and Sally Hemings, one of the Black women he owned and who bore several of his children, occupies the core of the book, but Gordon-Reed manages to craft a complicated and often contradictory history that extends far beyond the tangle of race, gender, and status that marked the Jeffersons and the Hemingses’ commingled journey through US history.

3. Team of Rivals: The Political Genius of Abraham Lincoln by Doris Kearns Goodwin (2005)This book follows the intersecting biographical tributaries of the powerful, ambitious men whom Abraham Lincoln, the nation’s 16th president, was able to steer toward the rushing river of his own turbulent civil war presidency. Lincoln as political strategist and savvy tactician is the frame that Goodwin points up most dramatically. But the book also succeeds at conveying Lincoln as a beleaguered and empathic head of state whose mettle is tried time and again by those around him and news from the battlefield.

4. Personal Memoirs of Ulysses S Grant (1885-1886)Rightly considered by many historians and literary critics as among the best of presidential autobiographies, this book was completed a generation after the Confederate surrender at Appomattox as Grant succumbed to a slow strangulation by throat cancer in the 1880s. The memoirs provide a vantage point on the nation’s bloodiest and most defining conflict that only a soldier elemental to the war and its aftermath could offer.

5. Theodore Rex by Edmund Morris (2001)As the best biographical volume on America’s 26th president, Theodore Roosevelt, Morris’s book draws bold-coloured portraits of outsized historical figures, with equally knowing shades of nuance and frailty. Morris has the contextual eye of the historian and sets scenes that are alive and convincing. He also conveys mood and meaning as well as any novelist.

6. Franklin D Roosevelt and the New Deal, 1932-1940 by William Leuchtenburg (1963)Dated, frayed, and surpassed by newer research and more eloquent storytellers, Leuchtenburg’s volume on the first two presidential terms of Franklin Roosevelt still stands the test of time as a scholarly, well-researched, and jargon-free narration of arguably the most consequential presidency of the 20th century. It is the tale of the rise of the liberal welfare state against the backdrop of the Great Depression and the gathering clouds of world war. Leuchtenburg tells the story well and sets the standard for future researchers.

7. The Making of the President, 1960 by Theodore White (1961)White’s fascinating chronicle of the 1960 presidential race is the starting point of quality, book-length journalistic coverage of modern American politics. Writing in the moment, White had an eye for discerning the essential character of men such as John F Kennedy and Richard Nixon who sought the country’s highest office, even as the media ecosystem of his day made such discernment more difficult to achieve.

8. Sleepwalking Through History: America in the Reagan Years by Haynes Johnson (1991)Johnson captures the zeitgeist of the 1980s by juxtaposing the countervailing forces of American optimism – or the desperate need of many Americans to again believe in their scandal-wracked government – against the greed, corruption, militarism and debt that threatened to unmask the soothing myths of American exceptionalism. At the centre of Johnson’s story is a self-made man, an actor by training and temperament who through force of will, theatrics – and a good dose of luck – led the country through domestic and external perils whose ramifications are still being felt today.

9. Dreams from My Father: A Story of Race and Inheritance by Barack Obama (1995)Of Obama’s autobiographical writings, this one provides the best understanding of his origins and burgeoning sense of self. His early and more frank ruminations on race are present here, and the book is not encumbered by the exigencies of political campaigning. At once a memoir, travelogue and deeply introspec­tive meditation, it is a fluent self-study of his efforts to reconcile himself with his eclectic lineage and to discover his place and pur­pose in the world.

10. Game Change: Obama and the Clintons, McCain and Palin, and the Race of a Lifetime by John Heilemann and Mark Halperin (2010)The essential volume on the 2008 presidential primaries and general election. Heilemann and Halperin had generous access to many of the historical players – including Barack Obama, John McCain, Hillary Clinton and Sarah Palin – and their staffs. It is a fast-paced, even breathless read, and anyone who paid even casual attention at the time to the historic events chronicled here will recognise its richly drawn characters, plotlines and twists of fate.

• The Black President: Hope and Fury in the Age of Obama by Claude A Clegg III is published by Johns Hopkins University Press. To help the Guardian and Observer, order your copy from the Guardian bookshop. Delivery charges may apply.