I HATE HIS FAMILY 😡😡😡

Can someone please give me some real advice. I really really hate my soon to be husband family. Like every time we're at his grandma house & the whole family together it seem like everybody is talking about me or saying I wonder why they're together. But if it's just two or three members things are okay. We have a baby on the way & im starting to really regret it cause I don't want to keep my baby from them but at the same time I don't want my baby around them. I haven't did anything to them. I'm very sweet and well respectful & I think that's the problem cause I don't smoke and wanna turn up every night like his exs. I literally will come in speak & leave because I'm miserable there. I tried talking to him about it but all he can say is do I wanna break up. I truly love this man but his family has to get better. I'm just praying things change before the baby gets here. Plz leave advice on what you think I should do or how I can handle it better.