Debating if I should invite my in-laws.

Ok before you go judging first read my post. I've been married to my husband 4 years prior to us he had been married when we meet he was split up and not with her anymore still married but not with her anymore. when we meet in 2010 we talked for a whole year didn't date right away untill he got divorced. Anyway we dated after that for two years got engaged got pregnant with our first and got married we have 2 girls and baby #3 due in nov. we love each other but there's one problem his parents. They don't seem to acept me. We never get invited to holidays parties, bbq nothing really. I know they don't like me bcs they have never made an effort to get to know me. We don't vist them since my husband works so much and they don't visit us. Witch my husband never makes an effort either and I don't understand why he doesn't really care much if we see them or not. During holidays we always spend it with my side of the family. Never mentions about going over to his parents. So anyway it's sad bc my daughters are growing & don't really know my husband's side of the family. I have tried to get us to socialize but it seems like they just don't really care to much. And I feel bad bcs I feel I did nothing to them and in the long run the ones that will end up hurt are my girls. My 4 and 3yr old there such sweet girls and it hurts a little that my husbands parents don't seem to love my girls for whatever reason. My husband had a son and daughter from previous realationship and I feel like they think I took there dad away or something. I've recently found out from my step-kids there mom goes over to my in-laws sometimes and they have all gone camping and other stuff. It's fine if they really like her. But what I don't understand is why they don't like me. I recently found out that my mother in law and sisters in law have said mean things about me. Like that I'm a slut and I ruined a family and got with a married man. Which I know it's not true bcs when me and my husband meet yes he was still legally married but was separated and I know this bcs I talked to his ex. She told me she still loved him and wanted to work things out with him. I told her fine I'll get out of the way so they can work things out then she told me her self things were done he wanted nothing to do with her bcs she had an affair so there for I know for a fact I didn't take no ones man I made sure they were done so I don't understand why they would say that. And I've Hurd from ppl his ex goes around telling ppl I took her husband away when she told me herself what happend and why it was over between them. I don't know why his parents would say that if she herself told me she had an affair they must not know that about her. But feel like I'm missing something or am just jumping into conclusions. Me and my husband have fought over this bcs I feel my daughters have nothing to do with this. Whatever it is that maybe happend and they feel it's my fault. I also don't feel like we should invite my inlaws to my girls birthday party this year. I invite them every year and that's the only time they see them on there birthdays it's not fair, my daughters have nothing to do with any of this. They don't need them. If they don't want to be a part of there life's bcs whatever reasons they don't like me then so be it but I won't ask them to be around my girls anymore if they don't make an effort. Also when my husband mentioned to his mom i was pregnant she said and what happened like we made a mistake or something. My husband doesn't take us around them either so I don't really know what to think anymore I just know I don't want them at my girls party and have them pretend like there great grandparents to our girls when there not. My husband said if we don't invite them then my girls can't have a party. So what should I do should I stand my ground or just invite them. Deep down inside I want my girls to have a realationship with them but I don't see an effort on there part. And I know there great with the other grandkids I just don't know why they don't see my girls the same.