Should I quit my job or leave the father of my child?

I've been with the father of my child for 4 years and we are engaged. I have been at my job for 8 years. I'm a Youth Care Worker. I work at a youth home. My job usually consist of my monitoring the teens, taking them to school, to extracurricular activities, and being a support for them. My fiancè recently told me if we are going to get married I either need to quit my job or only work with the females because he's uncomfortable with me working with other "men". They aren't men. They're boys. And I told him I'm offended that he would 1. Think I would cheat on him and 2. See me as a predator who would abuse my position like that and prey on kids. He said he's not saying that but temptation is real and he says a real wife wouldn't put herself in that position. Only one type of people can be tempted by MINORS and that's pedophiles/predators. I'm also not allowed to say I will only work with one gender. I have to work with all the teens. So he's saying I should quit my job and go elsewhere where I won't work with "men" or boys. My mom thinks I should and relationships are about sacrifice and I can have any job but my son only has one dad and I shouldn't break up my family over this. Maybe I'm selfish but I really like my job...