You're wrong. Men consider themselves to have a dog in the fight is they believe trans women to be predatory men and therefore better placed to lay hands on 'their' women in ladies restrooms.
It's of course a nonsense but no-one ever accused men of making sense.