Father-in-law problems... odd right

Ok so my father-in-law has never actively been in my husband life. He was around but he wasn’t there if that makes sense. He never taught my husband anything, he never bought my husband anything, he never spent time with him or even attempted to go to his football games as a kid. He will tell him happy birthday only if my husband calls him and its always days late. Yet he has the audacity to get mad at my husband for not visiting and bringing the kids over. My husband is always tore up about this, he knows that it’s not his responsibility to have a relationship with him, and the kids as well. So yesterday he went over and asked him why he didn’t want to come around. Y’all this man said it was because he didn’t feel welcome because of ME! He said that he doesn’t feel comfortable around me. I have never done anything to him, said anything out of the way, I mean nothing! He hasn’t been in his life since he was a kid and now I’m getting the blame?! What?? I was so confused when my husband came home and told me that. I asked what I done to make him feel that way (so I could fix it or change it) and my husband started yelling at me! Like he had believed him, he manipulated him into thinking I was the problem. I was floored. Especially because I would ask my husband if he wanted to go visit with him, he would always say “No, he don’t want me around anyways” or “No, if he wants to see me he’d come over here”. I then got upset and told him that the only problem I’ve ever had with his dad was the way he treated him and how he’s never been there for him. I said I don’t know how you can sit there and tell me I’m to blame for your father not being apart of your life, he wasn’t before me, nothing has changed, he just has another reason not to be a decent dad and that’s not my fault. I don’t know what to do. I am a introvert and a very shy/quiet person, I don’t speak my mind often enough and I never start a conversation without being spoke to first. His family is loud, they say whatever comes to their mind no matter how awful it is. Basically they’re rude people. I don’t click with his family and I get that but that’s not a reason for them to not treat me as such. The treat me and my son different. We’re the outcasts. I try to look over it but this is my last straw. My husband is a very family oriented person and I’m afraid he might abandon us because he believes we’re the problem.