Mother in law!!

Jmbk
 Ok so me and my husband have been married almost a year now we've been doing pretty well but my mother in law hates me in a passive aggressive way. When my husband and I first got together she immediately hated me because I had two children and her son had none. It became this feeling of why are you with him is it for his money? Which totally wasn't the case (I have my own money) not that he's even the richest person in the world or extremely well off to begin with his grandmother was the same way even down to giving my daughter a dirty look one time so I completely cut them off and stopped coming around. My husband is the complete opposite you couldn't tell him these kids weren't his. Well now that we are married his family always wants us around and when I come around it feels awkward like I have to be on the defensive side when ever anyone says anything to me. They even want me to take care of  her grandmother now that she's sick (yes the one that gave my daughter the dirty looks) my kids don't understand it yet but I do perfectly well. They are 2 and 4 so I just don't want them around people who don't necessarily care for them. How do I get my husband to understand or am I being selfish.