trouble with in laws

My husband and I got married this past summer. Up until then I didn't have many issues with him family. Now that I am 18 weeks pregnant it feels like I am always the bad guy to his family. Our biggest problem is that his family always shows up unannounced to our house. They just walk in and then criticize me if our house isn't clean. I've gotten home from work (I get home first) and his mom is in our house. I've been sick and she's shown up and told me I'm sick too much and suck it up. My husband has never said anything, even when he's heard everything, and I just tried to brush it off. Lately it's gotten way worse. The last time his mom was over she commented that she hopes our child gets all of their families genes because I have food allergies and she doesn't want to deal with our kid being that way. And again, I'm sick too much and she doesn't think I have good enough genes. (I'm an elementary school teacher so naturally I catch everything) My husband was sitting right there and said nothing. Now, his brother is saying I'm not ready to be a parent because I didn't want to watch his kids today (his brother will show up and drop his kids off with no notice so we just mentioned not to bring them today if he was going to) because my doctor told me to rest today as I had some bleeding. His brother told him he's tired of my excuses and as a parent I won't get days off and I need to stop acting this way. He also brought up how they are getting annoyed that we ask what they are cooking when they invite us to dinner. We do this because I have a dairy allergy and we went to dinner and they made lasagna and I couldn't eat. So now we ask so I can bring my own food or eat before but his family says I'm being difficult and putting them out. Again, my husband said nothing. Now he doesn't understand why I am never home when I am just tired of feeling like I can't be comfortable in our house. We've talked about this so many times and he just says his family doesn't like when they don't get their way and to ignore it like he does. I come from a dysfunctional family myself so it's hard for me to feel like I don't fit in with any family. How can I get him to understand why I can't just ignore It?