You may well have a 'right' to apply that label, but that doesn't make it correct. And nor do you, by extension, have a right to act on it.

By analogy, if I say that 2+2 = 19, then by neutral definition I am playing that fact in one of a pair of binary sets, true and false. This is logical, but if you tried to assert that 2+2 ACTUALLY equals 19, you'd be laughed at.

And this is the point: You don't get to "freedom of speech" or "everyone has an opinion" something that is factually incorrect.

Anyone can have their opinions on trans people, that doesn't make them correct. And it certainly doesn't give them the right to assert that their opinion has more weight that the actual facts.