Support for the Conservatives in England fell by 11 points compared to 2022, when this round of local elections was last held.
The party lost more than half of the seats it was defending and fared particularly badly in areas where the Reform vote was highest.
It did make some gains, winning back its traditional strongholds in the London seats of Westminster and Wandsworth, which had been taken by Labour in previous elections.
Sir James, who lost out to Kemi Badenoch in the 2024 Tory leadership contest, said: “All of us, Kemi [Badenoch] included, would like to see progress happen faster, but remember, we’re less than two years from a really significant electoral defeat, and nobody is expecting that we turn that around overnight.”
He said he had “no doubt” his party would gain seats at elections next year.
“I am absolutely confident we’ll see the acceleration of support coming to the Conservatives, coming back to us from previous Conservative voters, and coming to us for the first time from voters who are looking at the alternatives.”
He accused the other parties, including Reform, of “doing this pick and mix of populist policies”.
“Reform don’t know where they stand on cutting the welfare bill, don’t have a defence spokesperson… they are not delivering a centre-right, right-wing position.
“What we’ve seen with Reform, and we will see that again, and we’ll also see this with the Greens, is where they do get elected they let the voters down and those voters start rejecting them.”
The Green Party of England and Wales made big gains, taking control of seats in London, Norwich and Hastings.
On Friday, party leader Zack Polanski said two-party politics was “dead and buried”.
“It’s very clear that the new politics is the Green Party versus Reform,” he added.