Help with baby momma drama. Really long but please help.

Me and my man have been together for 4 years off and on. He has a soon to be 4 year old child and the child's mother hates me. She talks crap about me infront of the child and tells the child she's not aloud to say my name in their house and that's she's not aloud to talk about me at all. I love the child like she would be my own. But my man and the child's mother have had encounters before while he and I were together but I forgave him and he hasn't done anything since. They never actually had sex while we were together but they've exchanged pictures and so fourth. I love this man and we all have done things were not proud of. Every time we would break up we would end up right back together. He is an amazing man and we've lived together for 2 years, has our own house, both of us have job and he us bought me a car that he pays the payment on. I'm having a problem just letting go of things when it comes to the child's mother. He went to go get his child Sunday so we could spend our day with her and he child's aunt told the child we were having a party and that there was a lot of people at our house trying to get the child not to go with him. (The child doesn't do well around a lot of people) but it took him 2 hours just to get the child to come with him. While he was trying to get the child the child's grandmother was telling my man that he needed to get back together with her daughter and all of that. Well to me that's extremely disrespectful so I messaged the child's grandmother on Facebook and told her   How he's not going get back with the child's mother because she's had her chances and that he's mine now. I could understand if we were just "dating" for a few months but 4 years? They asked him to fix the floor in their house because the board is dry rotted or something and I don't want him fixing their house because our house needs work too. He says he just wants to fix it so his daughter doesn't fall thru the floor. Granted I don't want the child to do that either but they can find someone else to fix it. It hurts my feelings how pretty much when the child's mother says jump he jumps. But anything I say doesn't matter. I know he isn't cheating on me but is there a way to relax and not argue with him about her so much? I can't keep on with this baby mom crap. I wish he would tell them that he's not leavin me and to stop talking bad about me. Like I wish he would see things in my eyes for once and take up for me while he's there to get his daughter. I know this is long but I need help. I can't loose my relationship over her.