should I?

LONG POST AHEAD.

so me and my sons father broke up about 2 months ago. before then he used to hit me while I was pregnant. not alot maybe 2 or 3 times. not to mention was extremely emotionally abusive and very manipulative. so after I had my son not much changed only got worse. I was providing for him working almost every day. he would watch him while I was gone but other then that literally did nothing. didnt get up in the middle of the night nor wake up with him and when I got home from I would have to wait on him hand and foot and take care of my son. when we broke up we got into a huge fight cause he was saying hes going to take my son away from me and I said over my dead body and I told him to leave. he hit me then left and texts me maybe every 4 or 5 days to "check up on him". he asks how he is and then never replies after that. am I wrong for not letting him see him? I brought him to his house once and the whole time he was like "why are you staying" "you dont trust me" "you're a dumb bitch" etc etc. his temper is way to short and would even yell shut up at my son when he would cry. I keep asking him to sign over his right and he just wont. he has a job doesnt give me money hes offered but then threw it in my face before he even gave it to me so I didnt even take it. I just want him to sign over his rights and just to leave us alone. I have his number blocked but I just cant help but feel like a bad person from keeping my son away from him. but I just literally CANT trust him.