My daughters father side

I kinda need to vent ; I have a 6 month old and another on the way, I really don’t feel comfortable having my daughter involved with her dads side of the family like his mom or sisters.. his mom is crazy she smokes weed a lot and she’s always negative and has said rude things to me in the past before I don’t want my daughter around that... I’m not sure how to bring it up to her dad because he’s a big mommas boy.. his mom is very controlling and thinks she knows best for her 24 year old son. She’s evil. When my daughters father and I weren’t together she had told me she was going t take me to court because I didn’t want her up at the hospital because a few days before giving birth to my daughter she had said I was a bitch and a slut and trapped her son and was going to ruin her sons life... I don’t want my daughter over at his moms house because one she smokes weed has people coming in and out and house is never clean and dirty and always cold.. idk how to bring it up to him maybe I seem selfish but I’m a first time mom and his mom drives me crazy ...