I don't want in laws around my baby.

Am I wrong for not wanting my fiancé family seeing our baby? The mother & grandfather mostly. It started with his grandfather. I have an auto immune disorder & a very serious disease which makes it where I get sick really easy & being pregnant already worsens your immune system. Well we had no choice but to live with my fiancé grandparents we've told them about me getting sick easy & how I can't be under a lot of stress since my pregnancy is a little high risk. The grandpa hasn't changed a thing. He still gripes about the smallest things to us, he neglects cleaning after the animals, which I'll find pieces of cat poo on the floor & he still has yet two weeks later to clean it up. It shouldn't be our job to show him what he needs to clean or to clean after his animals. I get it it's his house but I feel like he doesn't care about mine or the baby's health. Then my fiancé mother. I feel like she has no respect for my fiancé or me. She wanted to throw us a gender reveal party which we were fine with but she wanted to do the gender reveal & baby shower at the same time. My fiancé had a talk with her the first time & him & I both thought it was settled that we were to do them separate since we want gender specific stuff & don't want everyone to have to wait that long to find out. So it comes to about a month later & I have him text her so we can start planning the party. She tells him she's not doing it till two months later with the baby shower. He told her no we want to do them separate & have the gender reveal party with family only. She kept persisting that you have to have games with both parties & that they need to be together. We explained once again why we don't want that. She tells him to just stop cause she's doing it. So we told her we don't want to & just reveal the gender now. So everyone found out the way we didn't want them to. We think it's very ridiculous how she's acting & I'm afraid she's going to be even more crazy & over step her boundaries when the baby is here. There's other reasons why I don't want them around but this is just the tip of the ice burg. So sorry for such a long post but I need advice.

Edit:

Thanks for the advice so far. Luckily we'll get our place soon in about a month. Where we live it's such a small town & there's a lot of people living & moving here so housing is very limited.