Living separate taking a toll

So here's the scoop on my situation..  my boyfriend is 26 years old, divorced for over a year, and has 2 kids.  I am 23 and have no children.  We have agreed that we want to have children together and have recently started TTC.  We have been together for a little under a year and he basically lives at my house on the nights that he doesn't have the kids.  His ex wife doesn't know about me and the children only know me as his friend.  He says it's too soon to tell his ex about me because he doesn't want to upset her and have her withhold the children.  He is with me almost every day and we are on the phone all the time that we can when he isn't at my house so I know there's nothing going on with the ex.  It's just so upsetting that he lets her control his life the way that she does.  At this point it is stressing me out with having to deal with him living between homes.  His children love me and he basically lives here so I don't see why he can't let himself move forward so we can start our lives together.  At some point the ex has to let him live his life.  We are fighting over the situation a lot lately and it is taking a toll on me.  Anybody in a similar situation?  Any words of support?