Should he go?

Firstly I apologize for the long post!!

It’s my husbands nans 80th birthday meal next month which I don’t feel like I can attend as his family dislike me and have done for years so I’ll explain why....me and my husband got married 6yrs ago, we were both young and couldn’t afford a big expensive wedding so we just invited our parents and their partners and his nan. My mother wasn’t speaking to me at the time as she didn’t get on with my husband but still came to the wedding and brought along my brother and his gf which I didn’t know about until I was walking down the aisle to get married, my husbands family think I invited my brother and have not liked me since even thou my husband has explained to them on countless occasions that I knew nothing about it until that day. My husband has a brother too and he’s not bothered in the slightest about this but they still have an issue. They still don’t really speak to me and slate me behind my back to people that know me.

Fast forward to now...his nan lives in a residential building with other elderly people and I have my own business and have clients that live in her building, for the last 4 months she’s told my clients lies about me which has cost me their custom.

His mum and dad tell him he can do better than me all the time and that he should leave me even thou we’re very happy together and have children, every opportunity he visits them alone they have something bad to say about me.

Am I being unfair in saying to my husband that I think he should back me up on this and not attend this meal with his family? I’ve literally done nothing wrong to these people and have gone out of my way on countless occasions to help them (I take his mother shopping as she does t drive, visit his nan when I was seeing clients in her building g to make sure she was ok, raised charity money for his father) just for it to be thrown in my face.