Am I wrong ?????

My baby's father says he wants to take me to court because I won't let him see our two kids. I don't consider us in a relationship and plus HE DOESNT SUPPORT OUR KIDS IN ANY WAY !!! We have a one year old and 8 month old and I've really had enough of his s*** !!! He has not ever had a job !!! His mom always takes up for him and says as long as she buys what our kids need shouldn't matter to me !! IT DOES MATTER THESE ARE HIS KIDS NOT HERS !!!! So she cussed me out and he keeps aggravating me about him seeing our kids I said no because I'm tired of what he does to me. Also when I was pregnant both times he never offered to take me to the doctor and I always invited him to come see the ultrasound sessions he said no because he didn't want to. I never had any good support from him at all. When I was in labor both times he was never there at all and he didn't wanna sign the birth certificates. I just cried BOTH times while I was at the hospital because the other patients in the maternity ward baby's fathers were there at all times. I was so lonely. And he never wants to take them to the doctor when I had school. So he told his mom to do it and she was so happy about it. I'm just so tired of his bulls**t !!! I'm very stressed out over this Because I know it's my job to provide for my kids but at least have some type of help from my kids sorry father !!! I need advice please😔