SO family issues

During my relationship my SO mother passed away and this was very hard on him. Being the relationship we have he clinged onto me during that time and after, we grew even closer than before. But the issue is his family, after this it's hard for him to go around his family because it upsets him and makes him sad, and he feels they don't understand that. Instead they point fingers at me and blame me and say I'm the reason he doesn't call, come around as much as he used to. And it pisses me off 1) because when he does call or visit they bash him for not doing it more often, Which hurts him. 2) because Im the one who keeps track of birthdays, holidays, ect. To ensure that he calls and visits. I often remind him or even most of the time I'm the one convencing him to call in the first place. I think it's absurd to blame me and try to cause separation in his life when he's already been through so much. It upsets me because I don't want to have issues with my future in laws. And it's hard on me because this is the family I'm marrying into, that my kids will have a bloodline to, and it hurts to know they dislike me so much. At one point they were even trying to break us up. What is some advice you ladies can give me to fix our relationship with his family?