Mr Smith, whose staff have been supporting the asylum seekers, said the alleged violence left many “too worried” to go outside.
“It’s a pretty depressing situation to find yourself in, when you’ve already been through significant terrors elsewhere in the world,” he said.
“Some of the residents have told us how they’ve been chased by the far-right when they simply went out for a walk or to buy milk.
“It reminded them of the reasons why they’ve been forced to leave their home countries in the first place.”
The charity boss said an 80-year-old worker had a brick thrown at him during an episode of disorder outside the hotel.
Asylum seekers have been housed there since 2020, it is understood.
But on Thursday evening, Epping Forest District Council voted to urge the government to close the hotel, which the BBC understands is housing about 140 men.
The venue has been thrust into the national spotlight after a man living there was charged with sexual assault, harassment and inciting a girl to engage in sexual activity.
Hadush Kebatu, 41, from Ethiopia, has denied the offences and remains on remand in custody.