More than 200 people have told health bosses they would choose not to have specialist treatment if a cancer centre relocates.
They were responding to a consultation on proposals to move Mount Vernon Cancer Centre from Northwood to Watford.
The new facility would form part of the planned rebuild of Watford General Hospital.
One patient said the last thing people with cancer needed was difficulty in accessing treatment.
Mount Vernon Cancer Centre offers care to about 13,000 people across the region.
But health bosses say modern cancer care requires other facilities, such as accident and emergency and critical care, to be available on the same site.
The Northwood campus has no such facilities so the plan is to move the cancer centre to Watford General Hospital, as part of its rebuild project.
A two-month consultation exercise revealed nearly half of the 2,222 respondents supported the move, while 42% opposed it.
A total of 12% (267 people) said they would “choose not to have specialist treatment if it meant travelling to Watford General Hospital”.
A report on the consultation said the figure should be “interpreted with care” as it was a “hypothetical question about a proposed future service model”.