Selfish-ness?

Would I be selfish if I leave my bf of 4 yrs for putting out relationship aside just so he can build his family a home? For the past year and a half he's been hinting me that it's going to happen "soon". Were gunna be living together. But all of a sudden he dropped the ball and said he's going to build them a house first and then he'll work on out relationship. And everytime I bring up that he doesn't wanna be with me he just tells ME to leave him. And we haven't been taking for a couple days, and he seems fine. Should I let it keep going on like that? Idk what to do. I'm so confused.