Anti-immigration protester Sean Robins, 65, told BBC West he felt unfairly tarred with a far-right brush.
“It’s an utter smokescreen. I’m not far-right because I have a social conscience,” he said.
“But part of that social conscience is not wanting all these young men coming over here and ripping us off.”
Online publicity had suggested the march was organised by Britain First, but there was little visible evidence to suggest this was the case, including Golding’s absence and a lack of signage.
Somerset Council leader Bill Revans and Taunton’s Liberal Democrat MP, Gideon Amos, took part in the counter-protest.
“[I’m] absolutely determined Somerset should remain a welcoming place for people,” Revans said.
Amos added: “They don’t represent what’s best about Britain, they don’t represent what’s strong about our community, which is the fact people who’ve come here from all over the world have made Taunton a stronger place.”