Kids, bathrooms, public places

I'm curious, at what age do you stop bringing your son, into the women's restroom? My son just turned 5 and he is afraid of the hand dryers, and automatic flushing toilets, so he refuses to go into the men's restroom alone, and wants me to continue to escort him into the women's, but I don't want to make other women uncomfortable with him being in there.... And I also want him to overcome the fear of going in alone.... ??