Work Advice

I just recently got offered a position at a gym. Please do not think I am stupid for this, I am honestly going through a hard time.

Anyway, I am talking to a guy, who I really like. I mentioned that I got an interview and he said "oh, a lot of guys will probably try to talk to you." He works independently so he doesn't work around anyone. My problem is that now I feel like if I take this job - and work around other guys, there will be an issue. First of all (due to personal preference) I don't like working around a lot of other guys anyway because they make me feel uncomfortable. But now, I feel like I have to worry about this guy getting upset that other men will be working at the gym as well. Again, I know this sounds ridiculous to some people but I was severely brainwashed and am having trouble coming to terms with what is right and wrong. Should I be concerned that things will be rocky between us? He does sports and women are on the same team so it's kind of like the same thing I guess. Maybe he's just a little jealous? I know I would be if our situations were reversed. I even get jealous about the girls on his team. I want to look for another job, one that fits me a little better, but I can't really afford to turn this one down because I need a job.