confused

The father of my baby says that if I move with my mother to get away from him he would call the cops just cause he thinks his name is on her birth certificate he has all say in what happens and I don't..... he has called me names said mean things to me and it's a unclean environment for my baby but idk what to do cause I don't want him to call the cops and then tell me I don't have any rights idk what to do