Am I acting right?

My boyfriend and I have been together for two years and a couple of months and one of our biggest arguments are related to family issues.

He moved to my city over a year ago and my family opened the doors for him warmly. He stayed with us for more than 6 months until he settled, he was basically treated like another son.

His family never called to meet me or my family, they did not show any interest, to the point that it has been two years and I've only seen them once for a couple of hours and without an actual invitation, it was me who went there despite my parents wishes, because they are very traditional and did not like how my boyfriend's family acted with me, my parents took it as a refusal.

We have had a couple of arguments because he wants me to go stay there at their house sometimes and I normally say no for the previous reason and because his step dad has substance abuse problems and once got in trouble for touching his boss's daughter, which it was supposedly "untrue" but we never know. I don't know that man, I feel very uncomfortable sleeping under the same roof as him and when I tell this to my boyfriend he gets mad, he doesn't understand. I totally respect his family but for me they are like strangers and I feel like they don't really try enough to change that. I have tried but I feel like it is very one sided.

Please let me know if I'm acting selfish.