His grandma is on my nerves

Partner and I have been together 5 years with a lot of problems and breaks in between . We have a baby now who is 4 months old so things have progressively gotten worse with arguing and finances and decisions tbh . Well when I was 4 days pp my partner and his mother would bother me non stop to take the baby to go see his grandma who is doing great by the way when I made it clear I wanted to just bond with my baby and not have visitors at all until I’m somewhat healed and get the hang of being a new mom and with my emotions . Well the bothering me became so bad and no one gave a shit about my body in pain of having a newborn it was all selfish of them so I finally caved so I didn’t have to hear it anymore and took the baby over there just days after leaving the hospital. While I was there his grandma proceeds to say my child is dark and made a comment about how his mom had an affair and that’s why his brother is mixed blah blah . She’s very racists and conservative so it bothered me that she was implying my child was not his because she had jaundice 🤦🏻‍♀️😂like seriously . Pissed me off but I held my composure . We see her again just a few weeks ago and she’s making comments saying she wished the baby didn’t look like me and my partner just sits and lets it happen . I have voiced so many times how pissed and bothered I am that he can’t ever defend me or have my back but expects me to go sit at family dinners with this lady and act cool and take our daughter around her . It’s caused so many issues with us idk what to do anymore but the thoughts of leaving him have been frequent lately. Previously in our relationship he’s let his friends disrespect me and not step in and say anything as Wel so I just am feeling let down by someone who’s supposed to stand by me . He doesn’t understand why I can’t let it go and why this hate is growing for his grandmother . Any advice on how to go about this