Mother-in-law

Ashley
My husband and I got married last year but we have a 6 year old daughter and a 3 month old. Both of our parents are divorced and I have 2 (ex)stepdads that I am still in contact with because they raised me and they are more like a father to me then my biological dad is. 
But every single holiday we have to spend with his mother. We see her every week at church and she always wants us to come over. I keep telling my husband I do not want to spend every holiday with her because we have other family member and both of our dads haven't met our 3 month old yet. But he can not say no to his mother and it is ruining our relationship. 
What should I do? I have told him I don't want to spend every holiday with her so he end up going to her house by himself and says I'm just ruining the holiday and making it not fun and happy for everyone else.