visitation rights..single mom's?

Did u have to go to court to settle out visitation rights? if so what happen, did u guys agree to go together or did u have him do it? What if you approach it first to avoid all the drama would you have to pay? Also what days do they or did they give to your kids father? is it different for babies? I have a 3 month old and 18 month old. I am scared being a single mom I feel like I can't do it but he is always leaving and leaving me alone with the kids. all he wants to do is smoke weed & be with his uncle. yesterday his friends were outside in his uncle's card and I work at 5am til 12. I was exhausted and tired I asked him to stay I went up to his uncle cus my bf was still inside watching the kids (he only had to watch them for 2 hours because they wake up at 10-11 and itold his uncle if he can please leave that I needed his nephew to help me with the kids he told me I had to tell that to my bf and I walked away. I told my bf he has a choice n I was tired of him doing this. he still left and I was so angry. I called him and the kept clicking on me telling me to stfu and leave him alone. I texted his uncle and this what happen

he came back home 1 am. i had to get up at 5am for work. his sister who's still in HS offered to watch the kids but since I saw him I asked him, he said he was so I left. I come back home and I sit in the living room. I first said hi to my kids and walked away. 30 min later he walks in with DD and I calmly asked if what happen yesterday was right? he looked at me and told me "why the fuck you talking to me? we aren't together why are you even here" my heart dropped I asked him why was he doing this we eventually started arguing. I bought up his uncle and he told me it was my fault for hitting him up & that it was my fault he called me a bitch. I told him I fucken give up, it's not the first time this happens and he never defends me he never tries to protect me or stand up for me if he was done with me i was done begging for us to stay as a family. he told me to leave or he was leaving. I still followed him to he car I wasn't going to leave my kids and I was crying he was just screaming in my face telling me we were done and I gave him back the engagement ring. Its done I can't do this anymore. i. sorry for the long post. I come here from time to time for support and each time I get closer to feeling confident and walking away. I am.scared being a single mom