Former Labor minister Ed Husic has rebuked Anthony Albanese’s assertion that “good people” attended last weekend’s anti-immigration rallies, saying “I haven’t seen a good fascist yet”.

Asked on Wednesday if he agreed with the prime minister suggestion that among the neo-Nazis and far-right activists at Sunday’s rallies would have been “good people” concerned about immigration, the now Labor backbencher pushed back.

“Those rallies were whipped up by far-right extremists and neo-Nazis. A lot of people were warned about that,” Husic told ABC’s Afternoon Briefing program.

“I’m not in the business of doing the [there are] ‘good people on both sides argument’. I think a lot of people in the Australian public would have been extremely unsettled by what they had seen, the way in which people had been targeted, the way in which specifically Indian Australians have been targeted. I don’t think there’s any place for that.”

Sign up: AU Breaking News email

The former industry minister, who made history as the first Muslim to be elected to Australian federal parliament in 2010 and later federal cabinet in 2022, concluded his response to the question by saying: “I haven’t seen a good fascist yet”.

The comments are the first public display of the unease felt by some Labor MPs about Albanese’s characterisation of crowds at the March for Australia rallies.

Guardian Australia this week revealed the Labor MP for the Melbourne-based seat of Aston, Mary Doyle, sought to clarify Albanese’s language during a closed-door caucus meeting on Tuesday morning.

Speaking up in the meeting, Doyle used words to the effect of “bad things did happen at the rally, where do we draw the line?”, according to one Labor MP’s recollection of the question.

One Labor source said several other MPs also felt uneasy about Albanese’s language but chose not to speak up.

In his remarks to the meeting, Albanese reiterated to Labor MPs that not all of the protesters were associated with neo-Nazis as he warned against tarring all attendees with the same brush.

In an interview with the ABC on Monday, Albanese criticised the rallies for giving a platform to neo-Nazis who sought to exploit grievances about immigration to sow public division.

The prime minister also said there was “no doubt” that “good people” would have been among the crowds at the nationwide events, which called for an end to “mass migration”.

“There’s always good people will turn up to demonstrate their views about particular issues. But what we have here is neo-Nazis being given a platform,” Albanese said.

At Tuesday’s caucus meeting, Albanese said he was shocked to see far-right activists “openly in uniform” and said it needed to be called out.

He also addressed the apparent recruiting that neo-Nazi groups were conducting.

“We have to make sure we give people space to move away and to not push them further down that rabbit hole,” Albanese told the meeting.

Husic has broken ranks several times since being his dumped to the backbench in a post-election reshuffle, including to criticise the government’s initial response to Israel’s blockade of aid into Gaza.

Separately on Wednesday, Liberal Jacinta Nampijinpa Price sought to walk back inflammatory claims she made on ABC TV that federal Labor was promoting migration by specific ethnic groups, including Indians, to grow its electoral support.

Singling out Indians coming to Australia, the outspoken Northern Territory senator said sections of the community were concerned at “the core number, or the type of migrants that are coming in”.

“I think Labor like to be able to ensure that they’re going to allow those in that would ultimately support their policies, their views, and vote for them as well,” she said.

“This is Labor. Basically, it’s power at any cost. And we see that occur all over the place in terms of the way they conduct themselves.”

Indian Australians have been targeted by anti-immigration protestors, and in the wake of the March for Australia rallies on Sunday, federal politicians including Anne Aly and Julian Leeser expressed concern about racism directed at the community.

“One of the very clear calls to action [at the rallies] that was listed there was anti-Indian immigration, against people coming from India,” Aly told the ABC on Monday.

The shadow multicultural affairs minister, Paul Scarr, recently flagged the urgency of a more empathetic approach to migrant communities, including Chinese, Indian and other diaspora groups.

Husic, who appeared after Nampijinpa Price, noted Australia had a nondiscriminatory immigration system.

An hour after the broadcast, Nampijinpa Price walked back the comments in a statement.

“Australia maintains a longstanding and bipartisan non-discriminatory migration policy. Suggestions otherwise are a mistake,” she said.

“My remarks were made in a wide-ranging interview on ABC where I sought to highlight issues of uncontrolled mass migration and ruptures to social cohesion.”