A portrait of Margaret Thatcher has been moved from her former study in Downing Street, weeks after Sir Keir Starmer arrived in No 10, according to the new prime minister’s biographer.

Tom Baldwin said he believed the painting of Lady Thatcher, who served as Conservative prime minister from 1979 to 1990, was now hanging in another part of the building.

Downing Street officials said they would not comment on the building’s interior.

But the move has angered some Conservatives.

Former Tory Cabinet minister Sir John Whittingdale, who served as Lady Thatcher’s political secretary, said it was right for the former PM to be commemorated in the room that “was so much associated with her”.

“‘To [move the portrait] quite so quickly… it looks like playing to the Left, who do not share most people’s admiration of her and he’s probably appeasing his left-wing backbenchers,” he told the Daily Mail.

“But most people would recognise her importance to this country’s history and it is right that she is remembered for that.”

Former Conservative leader Sir Iain Duncan Smith called the move “petty”.

Former first minister of Northern Ireland and DUP leader Baroness Arlene Foster said Lady Thatcher – the UK’s first female prime minister – was “the most significant PM after Churchill”.

She added: “Not a good start from Labour, looks and feels vindictive and petty.”

However, Labour Education Minister Jacqui Smith defended the PM, telling LBC: “Keir Starmer, can’t win, can he? A few months ago, people were having a go at him because he said he thought he could learn from the leadership of Margaret Thatcher.

“Pictures of Margaret Thatcher will remain in No 10.”

She added: “I think probably Keir Starmer is more concerned about actually sort of cracking on with the job of getting the country to work properly than where the pictures are.”

The painting by Richard Stone, which depicts Lady Thatcher just after the Falklands War in 1982, was in her former No 10 study, unofficially known as the Thatcher Room.

It was commissioned in 2007 by former Labour Prime Minister Gordon Brown, who faced some criticism at the time from left-wing MPs for commemorating his Tory predecessor.

The £100,000 cost was covered by an anonymous donor.

In comments first reported by the Herald, Sir Keir’s biographer, Tom Baldwin, recalled a conversation with the prime minister in the study, when he suggested: “It’s a bit unsettling with her staring down at you like that, isn’t it?”

Speaking at a book festival in Glasgow, Mr Baldwin said the PM agreed with him and, asked if he would “get rid of it”, nodded in response.

In a social media post on Thursday Baldwin said: “I believe it’s hanging somewhere else in the building now, or that’s what’s planned.”

During this summer’s election campaign, Sir Keir praised Lady Thatcher as a leader who effected “meaningful change” and set loose Britain’s “natural entrepreneurialism”.

The Labour leader said that while he did not agree with everything Lady Thatcher did, she had “a driving sense of purpose”.