Is it bad that I don’t want to spend time with my boyfriend’s family?

For some reason I’ve never liked spending time with my boyfriend’s family. I really couldn’t even say why but I don’t like it. We’ve been together for 3 years now and have been living together for 1 but every time they either come over or we have to go over to his moms house I just dread it. They have never been mean to me or anything like that, if anything his mom has told me she thinks of me as her daughter and she has always been fully supportive of our relationship. Just recently we got into a huge fight because I didn’t want to go with him to visit a cousin that went to visit his brother a few hours away from us, i told him I didn’t think I was needed there and that’s why I didn’t want to go but he expressed how he just likes it better when I’m there and how he wants me to spend time with his family because it’s his family. He was openly hurt and close to tears when he was explaining this to me but I didn’t really feel anything about it. I tried to explain how it’s not my family and I’m not the person they want to see but he said that it hurts him to know that I think of them as HIS family because he wants us to be together forever and that will be my family when we get married. How can I overcome this? I have no feeling towards his family at all, I am not really close to my own family so I don’t care about his either. Am I wrong to feel this way? Is this something that can be fixed?