Fiance family still bf with ex girlfriend!!

I posted a while ago about having in law issues and here is my main one. His family (mom, sisters, cousins, even sister in laws) are all still best friends with his ex girlfriend. We been together for 4 years, we are engaged (wedding 5-25-15) and have a baby together and they still wont kick her to the crub and treat me as if im just a fling! It has gotten better, i set rules which made them hate me more. I made it clear that i, nor will my fiance or baby  be around any time she is around. Well this weekend we went to a party and his mother knew she was going to be there And didnt say anything to me. Point is, i want to keep away from them. Theres always drama and i know they just tolerate me, they dont really accept me. I just started talking to most of his family becuase of the baby. But my fiance is really close to them and we fight every time he wants to go over therr. I dont know if i can make this relationship work out if i deeply despise his family. I tried talking to him and them and they say i need to get over it, im the one hes with blah blah blah, but they dont see it as an issue. Anybody been through this and did it work or fail? Im actually thinking about putting the wedding off becuase i cant see  marrying someone whose family i cant stand. And no its not about being insecure or anything along those lines. not to be rude but i look way batter than her.