There is one phrase, one slogan, one promise which is associated with former Prime Minister Rishi Sunak more than any other.

It is “Stop the Boats”.

Yet, in his first wide-ranging interview since leaving Downing Street, the former prime minister says he regrets ever saying it because it was “too stark..too binary”.

And he concedes that it couldn’t actually be delivered.

This is just one of the “lessons from Downing Street” which the man who presided over the worst ever election defeat for the Conservative Party says he’s learned, in a conversation lasting more than two hours for my Political Thinking podcast.

It covers not only the mistakes he thinks he made, but also the disagreements about the right way to manage the economy that he had with Boris Johnson; the radical ideas he wishes he could’ve implemented; the lessons he learned from being chancellor during the Covid pandemic – and his attitudes to race and faith and Englishness as the first British Asian prime minister.

Sunak is in reflective mood talking about a job he was catapulted into, saying he didn’t “probably have the time to enjoy it in the moment or appreciate it in the moment because of the context in which I was doing it.”

That context was not just an economic crisis but a political one.

The Tories were on their third leader in just 50 days, and there’d been no election of party members or the wider public.

“I didn’t have a mandate,” he says, and defends his approach of trying to bring warring factions together.

To do otherwise, he says, “would have been a huge gamble because the thing just could have collapsed. And would that have been good for the country? I don’t think so. I think what the country needed was stability.”

He’s not changed his mind about wanting to deport migrants who cross the channel to Rwanda, and says he now backs leaving the European Convention on Human Rights (ECHR), if it’s not reformed.

He says the court has “taken on new powers. There’s been mission creep…It does need to reform or we should leave.”

When I asked him if he had taken his eye off the ball in regards to soaring levels of net migration, he concedes that although he “took very strong action to bring the levels of legal migration down…I should have done them sooner”.

Sunak was in many ways an accidental prime minister. It was October 2022, and he learned that Liz Truss had survived in the job for less time than that famous lettuce, whilst having a meal with his two daughters at TGI’s in Teeside after a game of bowling.

Four days later, he walked into Number 10 as her successor.

He says he “had very mixed feelings…given what had happened” but was driven by Hindu belief in dharma which he says involves “doing your duty”.

“You’ve just got to focus on doing your best, doing what you’re there to do, and not worry about the rest,” he says.

“It was a very helpful concept for me…I kept coming back to that. I said, ‘look, this is my job. This is what I’m here to do. I’m well-placed to try and solve the economic challenge that our country is facing’.”

Bringing the economy back under control after the markets panicked in the face of Truss’s unfunded tax cuts – or what he calls “fantasy economics” – is the achievement he’s clearly proudest of.

However, he spells out for the first time the scale of the disagreements he had not just with Truss, but with Boris Johnson, who he served as Chancellor.

“He and I had quite different views on economic policy. I’m a small state conservative. I believe in prioritising, trying to restrain the growth of public spending, being careful with our borrowing so that we can cut people’s taxes…he was less worried about those things.”

He tells me that at their regular Sunday night dinners in Downing Street they argued over what could and could not be afforded.

He says his worries were inflation and interest rates rising, because “when they go up it’s going to have a big impact on our public finances, because we’re going to have to pay more to service the debt that we’ve got”.

He adds: “We cannot afford to keep spending and borrowing at this rate, and that means you have to prioritise. We can’t do everything.”

Sunak insisted that any plan to subsidise people’s social care costs had to be paid for by higher taxes.

Now he thinks “we’re having another review now…I tell you, the answer is, do we as a country think it’s right to pay more taxes for a more generous social care policy? Yes or no? I personally think the answer is no.”

He believed in slashing billions from welfare bills, a more “radical restructuring of the state” to pay for increased defence spending, and he says he told Johnson that the UK’s net zero obligations were saddling the economy with cost.

Now he argues for abandoning the legal commitment to deliver net zero, made law by another Tory leader Theresa May.

During the Covid pandemic both Sunak and Johnson faced fixed term penalties for breaking lockdown rules.

Sunak tells me he thought long and hard about resigning after that but says he had a job to do and he clearly can still scarcely believe that he was fined for turning up at a work meeting early where a cake was produced for the prime minister’s birthday.

Much more interesting is the lesson he draws from that period.

We should all have been treated more like grown-ups, he tells me, and the public should have told that “even the scientists themselves are not united on this, or they don’t 100% know that this is the right thing to do”.

The long-term negative consequences of lockdown measures should have been spelt out because, he says, “we’ve seen the impact it’s had on school kids everywhere and the impact it’s had on their learning. And we probably didn’t talk about that as much as we could have done at the time.”

Sunak is proud to have been the first British Asian prime minister and talks movingly of the moment his grandfather – who’d been born poor in an Indian village – calling an old friend at home with tears in his eyes on his first visit to Westminster.

He’s angry too with those like a popular podcaster who declared recently: “He’s a brown Hindu; how is he English.”

“Of course I’m English, born here, brought up here,” he says.

“On this definition, you can’t be English even playing for England, let alone supporting them… I genuinely thought it was ridiculous.”

Here is a man proud of his roots and ready to admit mistakes, but who is wondering whether his rise to the top all happened before the country knew him, and could see beyond the super-wealthy Tory who was the fifth Tory leader in just six years.

“It’s a lonely job,” he says, “because it is 100% only on you.”

Many were sure that he’d called the election early to head off to a new life in California.

Nonsense, he says, he lives here because it’s home and indeed has just set up a charitable foundation – the Richmond Project – named after the constituency in Yorkshire, which he’s still proud to represent in Parliament.