Am i wrong to not let him take the kids?

My kids father hasnt seen our 6 month old for 2 weeks because he decided to leave the house after a argument.this isnt the first time it happens but throughout the weeks i texted him asking if he can watch the kids so i could work and if he wanted to leave afterwards he could leave. I never got a message back or phonecall. I ended up losing my job because i couldn't find a way to work or take them to my moms. The first time this happen i blew up over 500$ on uber rides gettin to work or gettin my mom over my house to watch them. Im still paying for that til this day since i used a credit card. On wednesday a week a half later he finally shows up because i blew up his phone it was super late and he didnt wantt to talk. He stayed the night and in the morning saw our daughter Whos about to be 2. Our son wasn't there because my mom kept him for the night.he said he was sorry for leading me on the night before(yes we slept together and i was willing to forgive him and let him come back) he said he was leaving and blamed me and told me to figure everything on my own. We had agreed somethin out but i agrreed to it so hed leave me alone. I told my parrnts and closest friends about it and they all told me i was crazy my parrnts told me he doesnt deserve to take them (the plan was he would take them Sundays and Mondays) and how am i comfortable knowing he hasnt spend time or has been a father to them the past two weeks or moments when this has happen? They think i should allow him to come over and stay here for a few hours to see them but dont allow him to leave especially when he doesnt have a safe place of his own (he was staying at his friends ghetto apartment) he told me he was going to go back to his parents but he hasnt even spoken to them and has been dodging them too. My mom keeps begging me that a 6 month old is too little to leave for the whole day. My kids father also showed up last night at the house we were living and again he made me feel like there was hope he came because it was his birthday and wanted to spend time together but in the morning he was actin super shady. We ended up arguing because he said he only came to go out n eat with me and leave but didnt want me to get the "wrong impression" i broke down and told him i had enough n that ill get the rest of my stuff n leave. I told him i felt like i was falling out of love with him and if i moved on its because of him. He said okay and let me be. I texted him a few hours later because i was weak. I said nothing negative just pretty much how i was always there and never left him. I told him i wasnt perfect but everyday i tried for him. I let him do whatever he wanted. He didnt have to cook or clean and hardly worked. I let him go out a few hours everday to smoke with his guys and yet it was me??? I told him to fuck himself (well i didnt use those words) and i already had put him on child support. Its in the process. My only thing is everyone around me is telling me to not let him take the kids that he can come over and i texted him saying he cant take them sunday because i feel uncertain now i feel like the petty baby momma and hes goin to say im only doing this because he left. Can anyone please give me advice my parents r telling me this is what he wanted and hes not responsible