Rishi Sunak refused to recommit to a Tory manifesto pledge made by Boris Johnson to bring down overall immigration levels.
The prime minister said he does want the net migration figure to fall, but would not stick to the pledge made by the Conservatives at the last election.
Instead Mr Sunak said on Wednesday he had “inherited some numbers” on migration and insisted voters were focused on tackling illegal migration, such as small boat crossings.
His refusal to commit to the pledge came as Jeremy Hunt has insisted the government would take a “pragmatic” approach to immigration in the wake of the home secretary’s call to reduce the numbers coming to the UK.
The chancellor said ministers would be “sensible” about using migrant labour.
But he suggested that British workers would fill gaps in the economy over the longer term.
His comments come after the home secretary Suella Braverman said that the government “needs to get overall immigration numbers down”.
She also said Britain could train enough of its own fruit pickers to fill gaps farmers complain have left them struggling.
However, Mr Sunak said he would allow more seasonal fruit pickers into the UK if needed.
The 2019 manifesto promised that “overall numbers will come down” as the Government ends freedom of movement in the wake of Brexit.
At the time, migration levels were at 226,000 and it is expected that figure could have tripled when the latest Office for National Statistics figures are published.
Speaking to reporters travelling with him to a G7 summit in Japan, Mr Sunak said he was “committed to bringing down legal migration” but would not stand by the target.
“I’ve inherited some numbers, I want to bring the numbers down,” the prime minister said.
Pressed again if he stands by the figure, Mr Sunak said: “I’ve said I do want to bring legal migration down.
“I think illegal migration is undoubtedly the country’s priority, and you can see all the work I’m putting into that.”
The prime minister is under increasing pressure from within his own party on immigration, amid fears that figures released next week could show net migration at record levels.
Mr Hunt told the British Chambers of Commerce that Brexit, in which immigration played a major role, was a decision to “change our economic model towards being a high skill and a high wage economy” and to move away from “unlimited, low-skilled migrants”.
But he said that as well as reforms to help get the jobless back into work, there would still be a part for migrants to play.
“What I’m trying to do is to make sure that our businesses can find the labour that they need in order to make sure that recruitment isn’t a problem,” he said.
That meant “we will, at the margins, always be pragmatic” about areas where there are labour shortages.
But he added that if the UK had an employment rate in line with the Netherlands, there would be 2.7 million more people in work.
“So what we need to do is to tap into the incredible potential that we have at home, at the same time as we make that journey to be sensible and pragmatic about the immigration requirements,” he said.
Mr Hunt also insisted that the government was “completely united” on the approach.
Earlier this week Ms Braverman called for Britain to be “less dependent on low-skilled foreign labour” and claimed job shortages could be filled with domestic workers.
Several key industries are reporting staff shortages, including agriculture, food processing and logistics. Addressing a conference in Westminster on Monday, the home secretary insisted the government “needs to get overall immigration numbers down”.
“We mustn’t forget how to do things for ourselves,” Ms Braverman also told the National Conservatism Conference. “There is no good reason why we can’t train up enough truck drivers, butchers, fruit pickers, builders, and welders. Brexit enables us to build a high-skilled, high-wage economy that is less dependent on low-skilled foreign labour.”
