Locker room advice

Today after swimming with my 1 yo son we were both changing in the women’s locker room when a man wanders in and just start walking around. I’m in only a towel at the time, but can tell that this man has some sort of mental disability, so I go up to him and say “excuse me this is the lady’s room” he points to a woman so I ask her “is he with you?” She just ignored me. He wandered out, but then wandered back in a few minutes later. He ended up wandering in and out a few times while I was there. Now I did not report him to the front desk because I could tell he was disabled but it still made me uncomfortable to have a strange man in the locker room with me. Now our gym does have unisex changing rooms between the men’s and women’s rooms in the forms of single stalls. I choose not to use these because my son has crawled under the door to get out of them a few times. My question is, is there or was there something else I could have (or should have) done in this situation?