Sexist?

Is it sexist that I'm afraid of men. I'm a rape survivor and literally ALL men scare me that I know could overpower me. I try to make sure I'm never in a situation where I'm alone with a man but obviously at times it's inevitable. For example, I had a neighbor move in and they're both men and I ran into them going into my apartment at 10 pm. I told my husband (yes I'm married and yes I was terrified of my husband until getting to know him) that it terrified me and I had a panic attack once getting in the apartment. My husband perceeded to say that I was being sexist with my fear of men and it kind of made me feel bad.. I don't actually think badly of men, it's just I know (if they wanted to) they could overpower me and do what they want. Am I being sexist for feeling this way? Please no judgement.