“Don’t worry, Malcolm. The American people will never elect a lunatic to sit in this office.”

So Barack Obama had enigmatically assured me in the Oval Office in January 2016, when I asked him about the presidential race.

Well, it was now November and the unthinkable had happened.

And lunatic or not, Trump had won.

It wasn’t only Americans who were stunned. Nobody had expected a Trump victory and truthfully few were prepared for it.

He’d run a bombastic campaign, much of which seemed to us to be designed to ensure he wouldn’t win. What sort of candidate would refer to his opponent, a distinguished former first lady and secretary of state, with a nickname like “Crooked Hillary” and beam as his supporters chanted “Lock her up”? How could you get elected in (by Australian standards) prudish America when you have talked about “grabbing” women “by the pussy”?

We knew what he’d said in the campaign, but did he mean it? After all, everyone seemed to accept he didn’t expect to win. Was the whole campaign just an enormous exercise in self-promotion?

Every country and every leader tried to work out how to deal with Trump. Elaborate psychological analyses were written in foreign capitals – including our own. The general conclusion was that Trump was a narcissist who’d respond well to flattery. “Lay it on with a trowel” was the consensus, echoing former British PM Benjamin Disraeli’s advice on how to deal with royalty.

I felt this approach was quite mistaken. I’d never met Trump or dealt with him but knew plenty of people who had. He was typical of more than a few of the billionaires I’ve known – Kerry Packer, Conrad Black, Jimmy Goldsmith and Bob Maxwell, just to name a few. And the one thing I’d learned with bullies is that sucking up to them is precisely the wrong way to go.

Just as imperial powers regard deference as their due, so do bullies – especially powerful ones – expect to be flattered. It doesn’t win respect, nor does it earn gratitude. And if the bully in question is a particularly manipulative one, the flattery will be used against you. Personalities like that often appear utterly lacking in emotional intelligence, devoid of empathy. But that’s not the whole picture: in my experience, the successful narcissistic bully is able to manipulate others effectively because he has a keen sense of others’ vulnerabilities. Like any predator, he can sense fear and weakness from miles away.

So, the best way to deal with someone like Trump is to be frank and forthright. Be yourself, always be courteous – there’s nothing to be gained from rudeness or scratchiness. But stand your ground. That suited me.

There was a scramble to contact Trump once the election result was known. He had no transition team in place, but, thanks to Joe Hockey, I got a number from US-based Australian golfing great Greg Norman and I was able to call Trump on 10 November and pass on my congratulations. However, a major storm awaited us.

‘Then the sky fell in’

I’d agreed the refugee deal with Obama, in principle, at the White House in January 2016. It took some time to settle the terms of the deal and it was signed when I was in New York for the UN’s leaders’ week in September. As previously mentioned, both sides recognised we were better off not announcing it during the campaign lest Trump seize on it and make an issue of it.

So, the deal was announced on 13 November and the Department of Homeland Security got straight to work; 14 US officials arrived in Nauru in early December and started interviews. Naturally, we were anxious to ensure that the rightwing media in the US, especially Fox News, didn’t latch on to the deal and make it an issue, so we were careful to say as little as possible.

I spoke with Trump’s incoming chief of staff, Reince Priebus, on 7 December and he gave me every impression they’d stick with the deal, recognising the importance of the US-Australian relationship.

In late January, we learned that Trump was proposing a new executive order to put a ban on refugees from Muslim countries – and it was immediately reported on as scuttling the refugee deal. After a very helpful introduction from Australian-born Dow Chemicals CEO Andrew Liveris, I called Trump’s son-in-law and senior adviser, Jared Kushner, who conferenced in Trump’s immigration adviser, Stephen Miller, and pressed on them the importance of sticking with the bargain I’d made with Obama. They were sympathetic and assured me the new executive order wouldn’t prevent the refugee deal going ahead.

Over the days that followed, we reached out to everyone we could in the new administration. In a discussion with Julie Bishop, Vice-President Pence assured her that the Trump administration would honour the deal. National security adviser Mike Flynn gave a similar assurance to Justin Bassi, his counterpart in my office. We felt our frantic lobbying had worked.

Our optimism was reinforced on 27 January when the executive order came out with language that expressly enabled the administration to admit refugees otherwise prohibited by the order if “admitting the person would enable the United States to conform its conduct to a preexisting international agreement”.

Thanks, Jared!

Immediately, a call with Trump was set up for 9am Sunday, Sydney time. We’d been assured the refugee deal was agreed, so we worked on the other topics to discuss.

Then the sky fell in. Just as I got to the office, Mike Flynn called Justin Bassi and Pence called Julie Bishop – both with the message that the president had changed his mind. He wouldn’t, under any circumstances, honour the deal and would I please not even mention it.

We were in the prime minister’s office in Sydney, looking down over the harbour with all of its benign tranquillity. I feared the call was going to be a rough ride. And it was.

Trump sounded tired. He was friendly enough at the outset but his mood quickly changed. As soon as I raised the refugee deal he became angry, repeating again and again that honouring the Obama deal would “kill him” and “embarrass him”. He said, “I will be seen as a weak and ineffective leader in my first week by these people. This is a killer.”

I described how the deal was fair, benefiting the US too. And the people on Manus and Nauru were not a security risk. But if you come by boat, you can’t get into Australia. “Even if you were a Nobel prize winner,” I said. So you’re a new CEO: you inherit some deals you don’t like; you honour them, take a writedown and blame your predecessor.

Trump argued that, as a businessman, I must have torn up deals. I told him I hadn’t. I believed promises should be kept, whether by companies or countries. I didn’t make commitments lightly, and kept them when I did.

As his anger rose, Trump kept talking over the top of me, with more intensity; it was as though at times he was talking to himself or perhaps the people in the room, which of course included Steve Bannon, one of the deal’s fiercest opponents.

At one point, I looked up from the phone across my desk at [media adviser] David Bold – his face was white with horror – so I turned to look out the window instead. I had to get Trump to commit to the deal. Not only was it vital for our whole plan to resettle the refugees without getting the people smugglers back in business, an early rejection by Trump would be a political catastrophe in Australia.

So, I reasoned with Trump and finally won him over.

At the end of it all, I still wasn’t sure that Trump would stick to the deal.

The tone and the content of the call had been dynamite. We’d had nobody from my department or from the Department of Foreign Affairs present. We kept the transcript of the call secure and didn’t release it to the bureaucracy in the usual way. Not that we didn’t trust them, but at times you can’t be too careful.

Our plan had been to keep the whole thing very low-key. The deal was going ahead; Trump didn’t like it but he was a man of his word and he was sticking with it.

As the media interest grew, it appeared Trump was having cold feet as on 2 February he tweeted: “Do you believe it? The Obama administration agreed to take thousands of illegal immigrants from Australia. Why? I will study this dumb deal!”

Later that day someone, presumably in the White House, briefed the Washington Post about the call; it described Trump “blasting me”, quoted him telling me this was his worst call so far and that the deal I’d struck with Obama was the worst deal ever; it described his accusing me of planning to send him the “next Boston bombers” – although it omitted to mention my riposte that the refugees on Nauru and Manus, unlike the Boston bombers, weren’t from Chechnya.

The Post noted that the US embassy in Canberra had stated the White House had confirmed the refugee deal was going ahead “one hour and forty minutes before Trump suggested in a tweet that it might not go ahead”.

We assumed the leak had come from Bannon; his motive would have been either to produce enough heat to force Trump to renege on his commitment to me or, at the very least, to demonstrate to Trump’s base that he’d agreed to the deal only through the most gritted of teeth.

Back in Australia, I declined to comment on media reports about the call, simply restating that Trump had agreed to honour the deal and that the conversation had been frank and courteous. I expressly rejected the claim that he’d hung up on me – as the transcript shows he did not.

The leaking of the details and later the transcript of the call had the opposite of its intended effect. The refugee deal had never been a big news story in the US, despite the Australian media’s attempts to make it so. However, berating one of America’s closest allies was a big story and both fans of Australia and critics of Trump lined up to express their solidarity with us.

For our part, it was working out well. Trump and/or Bannon had over-reached and now we had the upper hand. More importantly, unlike my counterparts leading other countries, I’d got off on the right foot with Trump. We’d had a row, that’s true, but I’d stood my ground on a point of principle and he’d conceded.

An Intrepid meeting

However, we needed to ensure the personal relationship was as warm and positive as possible and so considerable effort was made to arrange a suitable visit for me to the United States to meet Trump for the first time.

The Australian American Association had been planning a dinner on the 75th anniversary of the Battle of the Coral Sea, to be held in New York on 4 May. The timing was less than ideal for me, just before the budget, but the setting on the USS Intrepid – where Australian and US veterans of the battle would also attend – and the occasion were both ideal.

The Battle of the Coral Sea was a major turning point in the Pacific war, when a combined US and Australian fleet turned back a Japanese armada heading to seize Port Moresby and cut off Australia from its US ally.

Jared Kushner told me on 1 May that Trump was still hurt about the refugee deal. “He hadn’t expected you’d come on so strong, although we knew that was your reputation.” I told him I didn’t think the call was so bad. “Malcolm, it was bad.” He told me Rupert Murdoch, whose father had founded the Australian American Association, had persuaded Trump to go to the function on the Intrepid, and he urged me to be low-key and deferential.

When Trump arrived, he was accompanied by Rudy Giuliani and Rupert Murdoch. His deference to Murdoch was greater than I’ve ever seen from any Australian politician and was in marked contrast to the high-handed way Trump treats most people. When he asked me if Rupert could join us for our bilateral discussion between leaders, I told him that wasn’t a good idea, adding: “The rest of the media will kill us; let’s catch up with him later.”

For a first meeting, we covered a lot of ground. He talked about Kerry Packer a lot; he knew I’d been Kerry’s lawyer and “kept him out of jail”, something Donald mentioned every time we met. Most billionaires keep an eye out for a good lawyer in a tight corner. He is a big fan of Greg Norman and described in detail several tournaments he felt Greg had unfairly lost.

Trump is a natural isolationist. Whether it was east Asia or the Middle East, Trump’s perspective was thoroughly dystopian. Everyone hated each other, had done for centuries and wasn’t going to change. So the less the US had to do with them the better.

After we’d had our one-on-one discussion and the media had left us, Donald suggested we ask Lucy and Melania to join us. Melania was found first and Donald described the refugee deal. By now he knew, just as I had told him, there were no security risks among the refugees.

“Melania, do you know, Malcolm has 2,000 of the worst terrorists in the world locked up on a desert island and that fool Obama agreed to take them. Can you believe that? And now Malcolm has talked me into taking them too! He got me to do something I promised never to do! He is a tough negotiator!”

Melania smiled, faintly and mischievously. “Just like you, Donald,” she said.

The subject of an incandescent row a few months before was now something to make light of. It was just another deal.

Steely resolve

The other significant problem we faced was trade.

When we met privately in Manila in November at the East Asia Summit, he asked me what I thought would happen if he banned all Chinese imports into the United States. “A global depression,” I said, as quietly as I could. He was clearly straining with frustration, and returned again and again to the “$350bn trade deficit, but it’s really more like $500bn, Malcolm”.

From my point of view, I could only do so much in trying to talk Trump around on trade generally – it was pretty clear that he had longstanding protectionist views he wasn’t going to change.

When he started threatening, early in his presidency, to take action to protect US industry by imposing tariffs on imported steel and aluminium, he was strongly encouraged by commerce secretary Wilbur Ross as well as his trade advisers Peter Navarro and Robert Lighthizer, who as far as I could tell simply wanted to use American muscle to extract better trade terms with other countries.

My singular objective was to protect Australia’s interests. If Trump was going to play havoc with global trade, I had to do my very best to protect Australia, to differentiate us so that we weren’t just another ally who was going to get run over by the Trump bulldozer.

So, after a lot of toing and froing between our embassy and the White House and between the various ministers and officials, it all came down – as it always does with Trump – to the great man himself.

After the love-in on the Intrepid, our next encounter had been in July at the G20 in Hamburg, hosted by Angela Merkel. She was a very courteous host but it was obvious she couldn’t stand Trump, and the feeling was mutual.

The challenge was finding a time to pin him down on the issue. He was certainly full of good cheer, inviting Lucy and me to ride in his car – “the Beast” – with him and Melania, and generally radiating bonhomie.

Trump was even good-natured about my supposedly “off the record” remarks at the press gallery ball in Canberra three weeks before. I’d given a rather Trumpian speech which brought the house down and, when it was leaked, went viral in the US. “So, you’ve been having a little fun at my expense, Malcolm?” Donald asked.

“Very affectionately,” I replied.

“It’s not bad.” He shrugged. “Lots of people think you are better than Alec Baldwin.”

Still, he knew I wanted a commitment on steel and he wasn’t going to make it easy.

Finally, we agreed to catch up after one of the plenary sessions and so we headed off to find a meeting room. On the way, Emmanuel Macron joined us, as did Theresa May. In vain, I did everything short of tripping them both to get them out of the way – whatever they wanted to talk about, I knew it wasn’t Australian steel or aluminium and I needed to get Trump by himself and extract a firm commitment.

Boxed in

Donald stopped and asked me if I’d ever seen his “skiff” – or that’s what it sounded like – and I could only say I hadn’t as I wondered why on earth he was talking about a sailing boat. “Oh, you gotta see it. It is so cool. Nobody, not even the Chinese or the Russians, can hear us in there. It is so secret!” Next thing, we took a right turn and found ourselves being led into a steel box the size of a container. All the while, Donald was explaining how utterly impenetrable it was to even the best bugs.

Of course, for the Scif (secure communications information facility) to work, you have to have the door closed. Even though it probably wasn’t a good idea to have a dozen or more people jammed into it, our trailing entourage piled in. With me were finance minister Mathias Cormann and our G20 sherpa – the senior Australian official guiding the G20 policy process – David Gruen. For Trump, there was at least Wilbur Ross, Steve Mnuchin and Gary Cohn. Emmanuel was there, as was Theresa, the German G20 sherpa and sundry other officials and diplomats.

Donald, Emmanuel and I were down one end of the Scif and, while I wanted to talk about steel, the president of France wanted to talk about the wording of the communiqué.

Donald wanted to talk about neither. “Emmanuel,” he said, “do you know Malcolm is the best lawyer in the world? The best. He kept my friend Kerry Packer out of jail. Nobody else could have done that.”

I was thinking, is this going to be a routine? So, on cue, I said: “Oh, Donald, it wasn’t that hard. He was innocent.”

Quick as a flash, Donald says with a big grin: “No, he was so guilty, so guilty. Deserved to go to jail forever!”

To say that Macron was astonished by this is an understatement, but the routine continued.

“Emmanuel,” Donald continued, “Malcolm has 2,000 of the worst terrorists in the world – the worst in the world – locked up on a desert island.”

“They aren’t terrorists, Donald,” I wearily replied.

“Oh, yes they are. They are the worst, and that fool Obama – the worst president EVER – agreed to take them to America. Can you believe that? Would you take them, Emmanuel?”

Macron opened his mouth but didn’t say anything; his eyes just got wider.

“But you know what’s worse? Now I have to take them! Malcolm, why do I have to take them?”

“Donald,” I said, “they’re yours. And they are very nice people. You will get on fine.” This was definitely becoming a routine.

To get to steel, I had to help resolve the wording issue with Macron. Trump quite sensibly didn’t really care what was in the communiqué as not only was he never going to read it, neither was anyone else.

Resolving the wording issues got Macron and May out of the Scif so we Australians were able to get down to the purpose of the meeting, which was to extract a commitment from Trump that he wouldn’t impose tariffs on our steel or aluminium. I was particularly concerned about steel because the tariff was higher (25% versus 10% for aluminium) and because our steel industry had been struggling in the face of aggressive import competition from China. Add to that, the second steelmaker, Arrium, based in Whyalla, had gone into bankruptcy.

The setting of the Scif may have been bizarre, but I had to fight for those Australian jobs. I explained to Trump that almost all of the Australian steel exports to the US were of rolled coil made in Port Kembla shipped by BlueScope to Steelscape, its subsidiary in California. Steelscape turned the steel into roofing materials – mostly Colorbond. Trump’s eyes lit up. “Colorbond, comes in bright colours, right?” – it turned out that he was familiar with it. “We used the gold-coloured one in some of our own projects. They really loved it in Brooklyn.”

Why did California need Australian steel? Well, as I described to Trump, the cost of shipping steel across the Pacific was about $40 a tonne versus $100 a tonne from the midwest or east coast steelmakers. So, I argued, if he put a tariff on Australian steel, all that would happen is that the cost of roofing steel on the west coast would go up by about 12.5% with no benefit to US steelmakers – the only winners would be the US Treasury and people who sold competitive non-steel roofing products. And the big loser would be the construction sector and American jobs.

“Donald,” I argued, “you have a great trade deal with us – no tariffs, no quotas and a huge trade surplus. So why would you want to hit us, just because you can?”

Well, we came out of the Scif with a commitment from Trump that there would be no tariffs on Australian steel. Ross argued that only the Colorbond product should be exempted and we discussed the practicality of that, but to both Trump and me that sounded like Wilbur trying to white-ant the deal before it was concluded, so from my point of view, the president had given me a commitment – no tariffs on Australian steel or aluminium, full stop.

I called him again on 10 March. We covered a lot of ground: regional security, China and North Korea as well as my concerns about how the United States and its allies had lost – and should work together to regain – leadership in wireless technology and especially 5G.

But my focus remained persuading him from imposing tariffs on our steel or aluminium. So I went through the big surpluses America had with Australia and how their trade deal with Australia should be their ideal. I sensed, as I had before, that he didn’t want to get pinned down to make a commitment, but I pressed on.

“Donald,” I said: “you can use Australia as an argument because you can say we have a level playing field, we have a surplus, so we aren’t going to impose steel tariffs.” He could, I argued, say to other countries: “If you want to have no tariffs, give us a level playing field like the Australians do. Fair and reciprocal trade.”

Trump liked the term “fair and reciprocal” a lot. That was very promising; that was a better theme for his trade policy than the American “might is right” message, which might go down well in the midwest but was so corrosive of American goodwill everywhere else in the world.

So in hands-on chief executive style, he started dictating a tweet, with me commenting on it as he went. How many years had we been an ally? Two hundred? “No, just one hundred to 4 July this year,” I said. And at one point we nearly had “the great state of Australia” in the message – “Perhaps nation,” I suggested.

It was a practical and good-humoured collaboration. We even compared our Twitter followers (he had 158 million, I had only 1 million), and finally settled on a very plain announcement saying that because of our fair and reciprocal military and trading relationship, the US would not impose steel or aluminium tariffs on the great nation of Australia.

And aluminium only just made it. I was so fixated on steel, it was only at the last moment I reminded him to include aluminium. No problems, steel and aluminium it was.

Trump’s tweet, when it came out, was slightly different: “Spoke to PM @TurnbullMalcolm of Australia. He is committed to having a very fair and reciprocal military and trade relationship. Working very quickly on a security agreement so we don’t have to impose steel or aluminum tariffs on our ally, the great nation of Australia!’

We were taken aback by the reference to a security agreement – it hadn’t been mentioned in the call with Trump – but we were assured both directly and via our embassy that this had been inserted by the White House lawyers, and that it didn’t detract from the president’s unequivocal commitment.

On 22 March, Trump’s proclamation on steel imposed tariffs on all countries with the exception of Australia, Argentina, South Korea, Brazil and the EU – pending further negotiations. It wasn’t the unequivocal exemption we’d agreed, but it was a step forward. The challenge now was, in the midst of all of the Washington chaos, to hang on to it.

When the protectionists advising Trump continued to push for a quota on Australia, I had another call with the president on 22 May to try to turn him around.

Trump started determined that we would have a quota, and his concern was that if Australia was exempted, it would create a precedent. So my job was to persuade him that the singular position of Australia meant that it was in his interest to exempt us.

I said: “When we last spoke we agreed there would be no tariffs on Australian steel and aluminium, we actually tweeted it and said that to the world, that was the agreement … This is really a political issue of principle, it’s not really an economic issue … People will say, why have you got no tariffs on Australia? Is it because of the military alliance? And you can say, no, it’s because it is fair and reciprocal, that’s my principle, I want a level playing field. The Aussies have no tariffs and no quotas on us.”

Once again Trump saw the logic in our case and agreed that not only would there be no tariffs on our steel or aluminium but no quotas either.

A giant personality

Ever since he was elected Trump has dominated the headlines in a way no other president has before. While many self-styled conservatives support him, he is not one of them. Conservatives defend and support established institutions. Radical, populist, Trump, the authoritarian iconoclast, takes them all on.

The economy, so far, has boomed under Trump. He has certainly delivered “jobs and growth”. However, looked at from afar, America seems more divided, angry and polarised than I can remember. Libraries are being filled with analyses of Trump’s wilful and intemperate nature, his “chaotic” administration, scandals and conspiracies, real and imagined.

Traditional political issues have been overwhelmed. The political debate has been distilled into one issue: Trump’s giant personality.

He says America is more respected than ever. It depends what you mean by respect. Strength is respected when it is matched with values consistently advanced. Around the world, Trump’s deliberate unpredictability generates fear rather than respect, anxiety rather than certainty. His view of the world is much closer to the “don’t tread on me” isolationists of centuries past than it is the neoconservatives who, after the fall of the Soviet Union, sought to remake the world in America’s image.

In our own region he has started, but not yet settled, a trade war with China. The Covid-19 crisis will no doubt further encourage protectionism. It seems impossibly far off at the moment, but at some point an expanded and enhanced Trans-Pacific Partnership that includes China and the United States should be the goal.

At summits in Singapore and Hanoi, he has feted and flattered Kim Jong-un, who hasn’t denuclearised and continues to test his missiles. So far, at least, the leader of the most powerful nation in the world appears to have been outmanoeuvred by the tyrant leader of a bankrupt slave state.

Trump is not a warmonger. And as he promised in his campaign, he is working to disentangle America from its engagement in the Middle East and Afghanistan, learning, as he has often said, that it is easier to get into these wars than to get out of them.

America may be stronger in economic and military terms, but its influence is diminished. In fact, under Trump, America seeks less influence, not least by rejecting many of the global institutions created by the US after the second world war.

Most consequential of all, I fear, will be his withdrawal from the Paris agreement. The world will struggle to meet the challenge of global warming without American leadership. And wherever Trump creates a leadership vacuum, others will fill it, often with values very different to our own.

But as the prime minister of Australia, my job was to protect our own particular interests and, as far as I could, influence him to act in a way that advanced our wider interests, in particular the continued commitment of the United States to our region. And in that regard, dramatic flourishes aside, he has not let us down. The “indispensable nation” is nowhere more so than in our hemisphere.

We had a row over the refugee deal, but it is being implemented. A large number have been safely resettled in the US after very careful security assessments. And we are fulfilling our side of the bargain too. He wouldn’t have done the deal, that’s true, but it was a fair one nonetheless.

Our only other contentious issue was the steel and aluminium tariffs.

There, defying the stereotypes, he was both thoughtful and good humoured. He started off with a different point of view, but listened to my arguments carefully. I didn’t solely base my case on our hundred years of mateship, or the Anzus alliance. A century of solidarity does matter a lot, and especially in Washington. But the US has a lot of allies. My arguments persuaded him because I correctly couched them as being in his interest.

If the Australian trade relationship is your ideal – fair and reciprocal, no tariffs, no quotas and, as it happens, a big surplus – then you should treat us differently to others where the playing field is far from level.

For all of Trump’s so-called madness, in my own dealings with him I found him no less rational than many other billionaires I have dealt with over the years. For all of our differences, as two businessmen, we spoke the same language.

• This is an edited extract from A Bigger Picture by Malcolm Turnbull, published by Hardie Grant Books, recommended retail price $55.