See, I was with you until you got to that last paragraph.

What you're basically saying is that X has a right to label group A as N.

But that would only follow for a completely neutral set. The set however is not neutral. Gender Identity is well researched and has been for nearly 100 years. Gender Criticals do not get to redefine trans and non-binary people simply because they have a difference of opinion.