In law problem

Ok sorry for the whole rant here. When we first moved out here 4 years ago, my relationship with my in-laws was decent. Hubbies father has a big mouth and thinks he knows everything. He's partially black and white and thinks everyone is racist towards him. He accused me of trying to get inbetween him and his sons relationship. He's made comments and called my family a bunch of racists. The other day my mil mentioned she needed to talk to me so I called their home and she was working so I talked to my fil and asked what as up.... he yelled and reamed me out for 10 minutes before I hung up. He said I made his wife miserable, I don't care about them, he has expectations for his daughter in law that I have to live up to. Then his mother talked to me on fb and mentioned how she feels left out and that I never make time for her. I had to lay down the law and say I had no issues with them so I didn't know where this is coming from. They have hogged me every family holiday selfishly... while I've only seen my family for 2 weeks in 4 years. How the hell am I the one who doesn't prioritize family? I took them off of fb and hubby doesn't want me visiting them since he's on tour. I don't know wth to do, at this point I'm done with trying to build relationships with them.