At what age would/should you make your son start using the men's restroom on his own?

Renee 👦🏼👼🏻👧🏻👼🏻👼🏻👼🏻👶🏻 • Stay at home mom. My family is my life.
My son is 5.  There are only a few places that I feel comfortable letting him go on his own.  The mall, Walmart, big stores, etc. I do not feel comfortable making him go on his own.  He's getting to the age that I feel like he may be too old to go with me.  What are your thoughts?