Need some serious advice

So my boyfriend and I we been together for 7yrs and we have overcome so many obstacles but there is just one that is just tearing our relationship apart. We have two daughters as well. But we live in two different households. He lives with his parents while I live with my mom with our daughters we use to live together in his parents house but then I moved out due to me having problems with his mom. When I moved out I thought he would follow along but he didn't he stayed at his parents saying that in my house it would be to crowed if he came along my mom lives in an apartment and my other siblings stay here as well but my mom gave me the master bedroom. We are going almost 2yrs now living like this. He comes over and sleeps over for maybe a night or two when he has off from work and then he goes back to his house. I keep telling him what would be the difference between him being here permanently instead of just leaving it's like we're playing house and confusing our children. I'm fed up with this living situation the only reason we haven't moved out on our own is rent is very expensive where I live and he just got a new job after quitting the one he had due to management problems. I am at the point of being so unhappy in our relationship there are good days but just all this just cancels out the good. I'm also angry because he comes to see us when he cans now with his crazy schedule sometimes he can't come alone. Like I said I'm starting to get fed up. I am thinking I messed up just now by telling him that im seriously over this relationship but that's because we had plans to go to my cousins birthday and he said that he was going to stay home and we will meet up later since his dad was gonna be alone like are you freaking kidding me I love his dad but I don't see his other brothers stopping there days with there families because there dad is going to be alone. He's a grown man he has health issues but he is still able to care for him self. I'm just angry that I see him for certain hours of the day and certain days and I'm pretty sure our children want to see him for more hours of the day too. I'm ready to just walk away. Sorry for the long post but I'm just angry and sad this isn't how I picture my life would be with kids. I just want my family together does this make me selfish