Misley Mandarin, a British-Chagossian who is part of the group, told the BBC the British would have to “drag me from my beach” and “kill me” if they wanted him to leave.
Mandarin, who calls himself the first minister of a Chagossian government-in-exile, said the islands “belong to us” and urged the UK to “let Chagossians come back to their homeland as British”.
Lawyers supporting the Chagossians say they are prepared to go to court to appeal the removal order.
The four are being supported by a crew on a vessel parked about 182m (200 yards) offshore from the Ile du Coin part of the Peros Banhos atoll.
They had sailed to the islands from Galle Harbor on the south-western coast of Sri Lanka.
Adam Holloway, the former Conservative MP for Gravesham in Kent, who has now defected to Reform UK, helped the Chagossians return to the territory and remains on the island to help build the settlement.
Holloway, a former army officer, said the deal to cede the islands to Mauritius was “completely crazy”.
“We’ve done this because Britain is about to make a catastrophically stupid mistake,” Holloway said.
“We are now in a world of great power play. The base at Diego Garcia is absolutely critical to the security of the West.”
One of the crew supporting the Chagossians said the former MP was hiding out in the jungle and has not been served eviction papers.
The removal orders were issued by a British Indian Ocean Territory (BIOT) immigration official.
Footage shows the official arriving onshore in an inflatable small boat and handing over the eviction notice to Mandarin at a make-shift camp.
The crew supporting the Chagossians said the BIOT patrol boat had trouble getting ashore and borrowed their dinghy to serve the removal order.