Help. I need advice please. EDITED

Long story short: my fiancé does not allow our 3 yr old daughter to be at anyone’s house without us. Well, At anyone’s house with boys to be exact. That includes friends or family. My mom and sister are livid over it. My sister has an 8 yr old and a 16 yr old boy. He does not want our daughter around boys that’s not her age. I trust my family and he does too, to an extent. He always says boys will be boys and sometimes they don’t even know when they are doing something inappropriate. not to say that’s not true but idk. And I do always here horror stories of family and friends, the closest people to you, doing the bad shit to the family. But my mom is hurt. My sister is hurt. I’m in the middle. Idk what to think. Sometimes I get upset at my fiancé because of this rule but I have other people saying I should always respect her fathers wishes bcuz he is only there to protest as much as he could and how long he can. Is my fiancé absurd? Or is my family wrong for not respecting us? This has put a rift between my mom, sister and I. Thanks for reading!

EDIT: we see my family all the time. We go around the boys all the time. She is allowed to see everyone. Her dad is absolutely not keeping her away from the family. He just does not want her there without one of us there. That’s it. My mom almost hates him for it. And it’s causing pain in my relationship with them. And frankly I’m kind of hurt that my sis and mom would put me in the position to feel like this instead of respecting the decision. Again, we see them all the time at my sisters and at events, but my mom chose to stop coming over our house because she doesn’t like my baby’s father right now because of this. They both have totally isolated him from them and don’t talk to him anymore.