Would aliens actually be weirded out by this quality of humans?
I feel like any sufficiently intelligent species living on a planet will have some degree of biodiversity on said planet. And the chances of something being made to be a poison/deterrent for creatures other than the intelligent species is probably a large one, because it's pretty hard for plants and animals to make a poison/deterrent that kills everything without also killing itself. So if there is a gap for itself, there is a gap for other life to coexist with the toxin. And that's before accounting for the fact that something can be safe at low levels, provide benefits/stimulation/good feels at low levels, and toxic at high levels.
So I'd think it would be pretty natural for intelligent life to consume things that are harmful to huge swaths of other creatures.