Transgender bathroom rights question

I already know this is ignorant but I'm going to ask anyway because I really am curious. I totally support trans bathroom rights. I believe that someone who is transgendered should be able to use whichever bathroom they choose. But I have one question: if a trans person passes as the gender they prefer (like if it really appears as if a biosex male or female is in the "wrong" bathroom), then why do they have to use the bathroom of their sex at birth if they aren't protected legally? I understand it can be dangerous for some trans people, but there have been some photos of female to male trans individuals using the women's restroom, who legitimately look like biosex males. I guess I don't understand how it would be a problem for them to use the men's or women's restroom if they genuinely pass as the gender they prefer.