If you were born female and transition to male, now you have a male looking person going into a women's bathroom. I promise you THAT will create more disruptions then just letting people use the one they identify with.
Btw, in my decades on this planet, I have yet to ever see a "not the correct gender" use the "wrong" bathroom. I have, however, seen plenty of women use the men's restroom at concerts because the lines were quicker.