I’m somewhat afraid of older men

It’s not some kind of phobia or anything, it’s just something I noticed lately. For some info, I’m 17. I think ever since I was little, but even more so as I grew older, I’ve always been spooked when an older man looks at me, even if they are just trying to be nice. I tend to freeze up a bit when they touch me- not like inappropriately, just a casual pat or an accidental brush. Maybe it’s just because they’re strangers? I’m not too sure. I don’t ever recall being sexually assaulted or anything along the lines of that. When I was a child I was always surrounded by older men because of my dad but they didn’t do anything to be either. I was catcalled when I was 14, but I wasn’t traumatized or anything like that. Am I just being wayyy to paranoid?