It's sort of beside the point, I think? Because litigating the morality of an argument over hypotheticals is a red herring to begin with. The real point is about the conditions those people were living under, the power dynamics in play, and the response necessary to secure liberation. What they did is try to put you firmly within a framework of idealism, which is about doing the "right thing" (as abstract moral principle) even if the circumstances surrounding it would tempt you to do otherwise. So in their minds, you affirmed that you are hypothetically morally bankrupt and would not do the abstract moral principle "right thing." Naturally, that's going to make them uncomfortable around you because idealism is all about what people are capable of and whether they are willing to strive to overcome their "base urges."
The question is, do you want to be friends with them? If so, you'll probably need to go out of your way to be more diplomatic about this stuff. If this were a question of organizing, I'd say, don't bother. Friendship and making sure you stay safe, that people aren't viewing you as some kind of loose cannon when you aren't, can be a little different.
We shouldn't have to go above and beyond to deal with people like this, and let's be real: sometimes we're not going to. The idealist position would say we should always strive to, no matter what. Well, sometimes it's just not going to happen. BUT, that doesn't mean you have to leave these situations to impulse either. What you can do is try to learn from it by reflecting on what about it didn't work, how you would like to present yourself and your views going forward, what outcome it is that you're even wanting. For example, are you wanting to vent? to persuade? To be the opposition when everyone is affirming the norm? Keep in mind the last one can be very difficult to do alone and it's easy to slip into defense mechanisms instead of keeping a clear head, especially when people are throwing nonsensical hypothetical gotchas at you or citing some of the same talking points you've heard a thousand times.