Time to move on?

Long post sorry..

Hi everyone maybe I know the answer but don’t to accept it:(

I met this guy on a dating appt about 2 years ago everything was perfect we even thought of having a family together. So let me tell you the story about a year ago we decided to have a baby not living with him yet cause he was building our house has a good job and so do I. So I thought okay will give him a couple of months and will be living with him before the baby was born, during my entire pregnancy was just me by my self so kept thinking it was due to all the hard work he was doing. We didn’t see each other often even got to the point of no sex at all I felt so un happy and lonely. Time passed and baby was born she is now two months the house is almost ready just a couple of things left. About a month ago I felt so insecure and decided to end the relationship due that we didn’t get to see each other and he would make excuses not to see me or to have intimate time. He’s living with his grandparents, only child and is building house behind his grandparents house. He was never very romantic or things like that when we first met but he would find the time to see me but nothing like big so I decided to move into the relationship with him knowing he was that type. Back to the story. So it’s been a month since I decided to break up reason to see if he would pay attention to me and make things worked out, I realized he didn’t seemed to care anymore so I decided to ask him about getting back together but he keeps avoiding the question or says will see:( he only sends 1 or 2 texts a day but to ask for the baby that’s all, he also blames me for not spending time with the baby when I have Her 24/7 and working a full time does not want to help out with the baby and gets mad if I ask for things but still wants to be around. Does this mean he has moved on and just wants to keep contact because of the baby? I’m so confused and it hurts because I didn’t wanted to go through this with my baby need help understanding if is me the one who’s wrong?