am i wrong to be over protective with my baby with certain people

Me and my boyfriend get into a fight because I don't let him take my son to his families house with out me being there. I tell him I don't feel comfortable with him there with out me . Because his family is a bit careless when it comes to watching kids. My sons cousin is 2 years old going on 3 and she is a bully always pushing him when he is playing alone and not anywhere around her. I get upset and don't let here touch him bcuz one careless act something bad can happen ... my boyfriend family thinks just because they are the aunt etc they can do what they want with my son and stuff but the thing is I am not a whatever  kind of money that let's just anybody watch their kid, and on top of that I know my boyfriend won't even be watching him the way he is suppose to because he will be to busy drink with his family . Last time I told him to watch him and I left him on the sofa and my son fell off when his dad was suppose to be watching him. They are always trying to tell me how to do things like let my son cry when he hits himself etc and I don't on top of that some of them don't even have kids and they try to make "jokes" about how I am with my son and the ones who do don't even watch their own kids and let's them do whatever they want and they are barely toddlers.