‘The Negro is not. Any more than the white man.” So wrote Frantz Fanon, the Martinique-born revolutionary and intellectual, in his 1952 masterpiece, Black Skin, White Masks. He was making an argument about the illusory character of racial categorisation. And, yet, more than 70 years after Fanon wrote those lines, they still feel unsettling, as if they are a challenge not just to racialisation but also to our identity, our very being. That they should do so exposes the deeply conflicted relationship we still possess with race.

We live in an age in which in most societies there is a moral abhorrence of racism, albeit that in most, bigotry and discrimination still disfigure the lives of many. We also live in an age saturated with identitarian thinking and obsessed with placing people into racial boxes. The more we despise racial thinking, the more we seem to cling to it.

This paradox is at the heart of my new book. Not So Black and White is a retelling of the history both of the idea of race and of the struggles to confront racism and to transcend racial categorisation, a retelling that challenges many of the ways in which we think both of race and of antiracism.

Most people assume that racism emerges when members of one race begin discriminating against members of another. In fact, the opposite is the case: intellectuals and elites began dividing the world into distinct races to explain and justify the differential treatment of certain peoples. The ancestors of today’s African Americans were not enslaved because they were black. They were deemed to be racially distinct, as black people, to justify their enslavement.

We think of race today primarily in terms of skin colour. But that was not how 19th-century thinkers imagined race. It was, for them, a description of social inequality, not just of skin colour. It may be difficult to comprehend now, but 19th-century thinkers looked upon the working class as a distinct racial group in much the same way as many now view black people as racially dissimilar to white people. Only in the 20th century, as the working class was drawn into the democratic process, and as the new imperialism redrew the “colour line”, did the contemporary understanding of race emerge.

Many today imagine, too, that identity politics is a new phenomenon, and one that is associated with the left. I show that its origins lie, in fact, on the reactionary right and its primary expression, long before it was called “identity politics”, was in the concept of race, the belief that one’s being – one’s identity – determined one’s moral and social place in the world.

If much of the history of race has been obscured, so, too, has much of the history of the challenge to racism. Until recently, those confronting inequality and oppression did so in the name not of particular identities but of a universalism that fuelled the great radical movements that have shaped the modern world, from anticolonial struggles to campaigns for women’s suffrage.

These struggles expanded the meaning of equality and universality. There has developed in recent years an impassioned debate about the Enlightenment, which both supporters and critics present as a peculiarly European phenomenon. For the one, it is a demonstration of the greatness of Europe; for the other, a reminder that its ideals are tainted by racism and colonialism. Both miss the importance of the non-European world in shaping many of the ideas we associate with the Enlightenment. It was through the struggles of those denied equality and liberty by the elites in Europe and America that ideas of universalism were invested with meaning. It is the demise of that radical universalist tradition that has shaped the emergence of contemporary identity politics.

There have always been identitarian strands among antiracists, from 19th-century “Back to Africa” movements to Négritude in the 20th century. Only in the postwar world, however, have they come to dominate and to be seen as progressive. The reasons lie in a myriad of social and political developments, from the erosion of class politics, to the emergence of culture as the primary lens through which to understand social differences, to the growth of social pessimism, that have helped marginalise the universalist perspective.

The embrace of identity politics by the left has ironically opened the door for the reactionary right to reclaim its original inheritance, allowing racism to be rebranded as white identity politics. We have come full circle: the politics of identity that began as reactionary claims about a racial hierarchy has been regrasped by the reactionary right in the name of cultural difference.

The following edited extract from my new book shows how the far right remade itself in the postwar world and how it has exploited the language of identity to pursue its aims. It shows, too, how mainstream conservatives have allowed far-right tropes to seep into our culture.

As reactionary organisations, which had enjoyed the limelight in the prewar years, were pushed into the shadows in the post-Holocaust world, many on the far right were forced to rethink their views of race, identity and difference. Alain de Benoist became a key figure in this rethinking, the founder of the Nouvelle Droite in France, and a philosophical mentor of the contemporary far right.

Benoist cut his political teeth within the traditional fascist milieu, most notably through the far-right opposition to Algerian independence. In the 1960s, after the French defeat in Algeria, he recognised the need to move beyond discredited arguments rooted in biological racism, and to engage in a cultural war to reclaim intellectual ground. In 1968, Benoist helped found GRECE, the Research and Study Group for European Civilisation, a thinktank to school the far right.

The Nouvelle Droite drew in part from traditional themes and sources. It proclaimed its hostility to the Enlightenment, modernity, equality, democracy and liberalism, and insisted on the importance of tradition and hierarchy. It found sustenance in the French reactionary tradition from Joseph de Maistre to Charles Maurras, and from German rightwing thinkers, especially the interwar “conservative revolutionaries”, such as Ernst Jünger and Carl Schmitt.

It drew, too, upon a very different tradition: that of the New Left that emerged in the late 1950s. From the New Left, the French New Right borrowed arguments about the significance of culture, its hostility to globalisation, its anti-Americanism and its embrace of the Italian Marxist philosopher Antonio Gramsci. Benoist took from Gramsci the belief that conquest of power comes only after conquest of culture. Liberalism was so entrenched that its values survived irrespective of who was in power. Anti-liberals, Benoist argued, had to fight battles not on the streets but in people’s minds, at the level of ideas, and of “metapolitics”. This he called the tactic of “rightwing Gramscianism”.

At the heart of Benoist’s philosophy was the abandonment of racial superiority in favour of cultural difference, and the reworking of the relationship between community, identity and diversity. “The true wealth of the world”, he insisted, “is first and foremost the diversity of its cultures and peoples.” It is in being different that a people finds its meaning and identity, both of which are drawn, indeed in certain senses are inseparable, from its culture and heritage. “Different cultures provide different responses to essential questions”; hence “all attempts to unify them end up destroying them”. It was a völkisch vision: “Everyone inherits a ‘constituent community’ which precedes him and which will constitute the root of his values and norms.” The individual “discovers his goals rather than choosing them”, and builds his identity through that discovery. So, “to find out who I am, I first have to know where I am”.

Such “ethnopluralism” seemed not to possess the taint of biological racism; but by fixing cultures to specific geographic locations and by insisting that to belong to a culture one had to be descended from the original inhabitants of that location, the Nouvelle Droite found in “culture” the synonym for “race”; a find later borrowed by many conservatives and “postliberals”.

Immigrants, Benoist insisted, must always remain outsiders because they were carriers of distinct cultures and histories, and so could never be absorbed into those of the host nation. Citizenship should be reserved for those who are “one of us”. Immigrants could – or, at least, should – never be citizens. Democracy only works where “demos and ethnos coincide”.

“We are Generation Identity… We have stopped believing that Khader is our brother, the planet our village and humanity our family. We have discovered that we have roots and ancestors – and thus a future. Our only inheritance is our blood, our soil, and our identity… This is not a mere manifesto, it’s a declaration of war.”

It was a declaration of war on a YouTube video. But for all its comically dramatic music and overheated rhetoric, the launch in 2012 of Génération Identitaire, or Generation Identity, marked an important point in the development of modern reactionary identitarianism. Ten years earlier, a group of French far-right activists, many linked to the Nouvelle Droite, had formed the Bloc Identitaire, which became the heart of a network of far-right identitarian groups and of which Génération Identitaire was the youth wing. The movement was banned by the French government in March 2021 for “incitement of discrimination, hatred and violence”. By then it had spawned a dozen other groups across Europe, and its influence had crossed the Atlantic, too.

The Bloc Identitaire drew on the Nouvelle Droite for both individuals and themes. Its key leitmotifs are familiar: opposition to globalisation, defence of ethnopluralism and white identity, hostility to immigration and Islam. The Identitarians feared that demographic change would sweep away white Europeans. “The cradle”, writes Adriano Scianca, a leading figure in the Italian identitarian movement, is “the most powerful weapon” and when “the baby cots are empty, civilisation dies”, an echo of future US president Theodore Roosevelt’s claim at the end of the 19th century that “competition between the races” reduced itself “to the warfare of the cradle”. For late-19th-century white supremacists, the declining birth rate of Anglo-Saxons created the alarming possibility of the only “true white race” in America being overrun by “the immigrant European horde”. A century later, the fear is of Europeans being swamped by hordes from beyond the continent – and in particular by Islam.

Gisèle Littman, an Egyptian-born Jewish woman who wrote under the name of Bat Ye’or (Hebrew for “Daughter of the Nile”), coined the term “Eurabia”. It described a grand conspiracy theory in which the EU, led by French elites, implemented a secret plan to sell Europe to Muslims in exchange for oil. Europe, Ye’or told the Israeli newspaper Haaretz, “will become a political satellite of the Arab and Muslim world”. Europeans would be reduced to the condition of “dhimmitude” – the permanent status of second-class subjects of Islamic rule. The Israeli historian Robert Wistrich dismissed Ye’or’s fantasies as “the protocols of the elders of Brussels”. In the wake of 9/11, however, the fantasies took flight, and not just on the fringes of politics. The mainstream British writer Melanie Phillips has become an advocate of the “dhimmitude” thesis, as have influential figures such as Niall Ferguson and Bruce Bawer in the US.

Generation Identity is no mass movement; membership of its various groups is tiny. Nevertheless, it has helped shape public debate, promoting an aggressive form of reactionary identitarianism that has percolated far beyond the far right. “Europe is committing suicide… by the end of the lifespans of most people currently alive, Europe will not be Europe and the peoples of Europe will have lost the only place in the world we had to call home.” That could be Alain de Benoist or Guillaume Faye or any number of Nouvelle Droite or Génération Identitaire polemicists. In fact, it is Douglas Murray, in the opening to his 2017 bestseller The Strange Death of Europe. Murray is a leading figure in British conservative circles, associate editor of the Spectator magazine and author of a string of popular books. He writes of “the replacement of large parts of the European populations by other people” and worries that “London has become a foreign country” because “in 23 of London’s 33 boroughs ‘white Britons’ are now in a minority”, again echoing Generation Identity.

The main themes in Murray’s argument were steeped in traditional racial thinking. The term “race suicide” was coined in the late 19th century by the American sociologist Edward Ross, and popularised by Theodore Roosevelt, to express their fears that Anglo-Saxons were being out-bred by inferior immigrants from southern and eastern Europe. The white supremacist Theodore Lothrop Stoddard warned in the early 20th century that the white ancestral “homeland” of the Caucasus had become a “racially brown man’s land in which white blood survives only as vestigial traces of vanishing significance”. The same was happening in Europe, too. “What assurance”, he wondered, could there be “that the present world order may not swiftly and utterly pass away?” These ideas were for much of the postwar era pushed to the racist fringes. Sustained by the Nouvelle Droite and Génération Identitaire, these fringe arguments have now become appropriated by many strands of mainstream conservatism.

The 2010s saw a series of books warning of Europe “committing suicide”, such as Thilo Sarrazin’s Germany Abolishes Itself and Éric Zemmour’s The French Suicide. Sarrazin, former SPD finance minister for the state of Berlin, and executive board member of Germany’s central bank, bemoaned the declining white population and the high level of immigrant fertility, the combination leading to Germany being both less intelligent, less moral and no longer Germany. For Zemmour, a television journalist who became a candidate in the 2022 presidential elections, Europe was committing “premeditated suicide”, the left having “betrayed the people in the name of minorities”.

The “Great Replacement” conspiracy theory, a staple of the far right, has also gained a foothold in mainstream conservatism. In 2011, the novelist and white nationalist conspiracy theorist Renaud Camus published Le Grand Remplacement in which he claimed that globalists had created the “replaceable human, without any national, ethnic or cultural specificity”, allowing “the replacing elites” to swap white Europeans for non-Europeans. He described non-Europeans in Europe as “colonists”, the “replacing elites” as “collaborationists”, and the process of replacement as “genocide by substitution”. Camus dedicated his book to the two “prophets” that had shaped his thinking, the British anti-immigration politician Enoch Powell and the French writer Jean Raspail, whose 1973 dystopian novel The Camp of the Saints tells of a fleet of immigrants from India overwhelming France, and its white population, and has become a cult hit for identitarians across the globe.

In Britain, too, similar fears have become part of the conservative conversation. Like Douglas Murray, the London-based American novelist Lionel Shriver fears the de-whitening of London and projects her version of replacement theory. “The lineages of white Britons in their homeland commonly go back hundreds of years,” she writes, and yet they have to “submissively accept” the “ethnic transformation” of the UK “without a peep of protest”. Westerners, she adds, are being forced “to passively accept and even abet incursions by foreigners so massive that the native-born are effectively surrendering their territory without a shot fired”. The distinguished economist Paul Collier is another figure apprehensive about “the indigenous British” becoming “a minority in their own capital”. Political scientist Eric Kaufmann thinks it legitimate to promote white “racial self-interest” and to use such racial self-interest to limit immigration, so that in a majority white country, immigrants should be mainly white to enable “assimilation”.

Identitarian arguments have become even more entrenched on the other side of the Atlantic, from the far right to mainstream Republicanism. The white nationalist and neo-Nazi Richard Spencer, who claims to have invented the term “alt-right”, replays many of the themes of reactionary identitarianism: white people as victims of cultural “dispossession”, immigration as a “proxy war” against white people. He advocates “peaceful ethnic cleansing” and the creation of “an ethno-state that would be a gathering point of all Europeans”, one “based on very different ideals than… the Declaration of Independence”.

The presidential victory of Donald Trump in 2016 provided new opportunities, as alt-right identitarians such Stephen Miller and Steve Bannon entered the White House. Even before the Trump ascendancy, conservatives were humming to many of the European refrains. In the question at the heart of Christopher Caldwell’s 2009 book, Reflections on the Revolution in Europe – “Can Europe be the same with different people in it?” – is embedded the idea that Europe was made by a particular group of people and that immigrants – different people – would undo it. He echoes, too, the claim that migration is a form of “colonisation” and that migrants come to “supplant” European culture. Caldwell hails Raspail’s The Camp of the Saints as capturing “the complexity of the modern world”.

After 2016, the Great Replacement theory became commonplace in Republican circles. “We can’t restore our civilisation with somebody else’s babies”, Iowa congressman Steve King tweeted. Fox News’ Tucker Carlson has constantly charged the Democrats with trying “to change the racial mix of the country… a policy [that] is called ‘the great replacement’, the replacement of legacy Americans with more obedient people from faraway countries”. Polls show that one-third of Americans and nearly two-thirds of Trump supporters believe in the Great Replacement theory and that a secret cabal “is trying to replace native-born Americans with immigrants for electoral gains”.

One of the ironies is that many of the conservatives who fret most about “white decline” are also among the most strident critics of identity politics. According to Douglas Murray, identity politics “atomises society into different interest groups”, and its “consequences… are deranged as well as dementing”. But not, apparently, when worrying that “Only 44.9% of London residents are now white British” or that Europeans are being driven out of their homeland. Taking part in a debate in defence of the proposition that “identity politics is tearing society apart”, Lionel Shriver argued that she had been a “fierce advocate” of the US civil rights movement because its goal was “to break down the artificial barriers between us” and “to release us into seeing each other not as black or white… but as individual people”. “The colour of my skin,” she added, “is an arbitrary accident” and “the boxes into which I have been born are confinements I have struggled to get out of and I would wish that liberation to everyone else.” Except, it seems, if you are a non-white immigrant. Then, the “arbitrary accident” of birth becomes an essential feature of one’s identity, the “artificial barriers between us” need to be recognised as insurmountable impediments to assimilation, the “confinements” of ethnic boxes maintained and people seen not as “individuals” but as “black or white”.

The reactionary right – Nouvelle Droite, Generation Identity, the alt-right in America – uses the language of diversity and identity as a means of rebranding racism. Many on the mainstream right rehearse elements of this rebranding, even as they castigate the excesses of white nationalism. Murray “unequivocally” condemns the “racism exhibited by people pursuing white ethno-nationalism” while also giving a nod to the Great Replacement theory and to the importance of whiteness. It is occupying the grey zone in which one can claim attachment to the moral framework of postwar antiracism but also maintain the freedom to replay perniciously racist arguments, helping to normalise them.

• Not So Black and White by Kenan Malik is published by Hurst (£20). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply