How to handle a relationship when your partner is trying to keep your lives separate?

I've been with my current partner for over a year now. Over the past few months he's keeping his family life separate from my/our life. He has a teen daughter that lives in another state and family that lives all over he US. When he visits his family or daughter he puts out relationship on hold. He often will go a week without out calling and sometimes won't text either. I've been trying to talk to him about how it's not fair that he keeps me out of his personal life. He keeps telling me that it's just easier this way and he didn't think it would be fair to drag me along with his family activities. He added that in a year or two it might be different. ( I don't really see how time is going to change this)

Has anyone else had this happen? Is there a way to work this out? I kind of feel like this is the Down hill movement of our relationship.