My special interests are very special to me (ugh, what a terrible pun. Defo not intended!) and I reserve sharing them for people I trust. That alone is a reason for me to mask.
It's actually harder for me not to mask than it is to mask a lot of the time, even though my mask is far from perfect. But I'm "out" at work and it's fine if people realize I'm masking.
I think for me donning the mask is like donning armor to protect things I care about from a majorly uncaring world, and if I unmask around you it's a sign of trust.