Single Mom, scared about the father

Question:
After the father of the child found out I was pregnant, he wanted me to abort the child. I didn't feel comfortable with that, so we parted ways.
Last night he went off on me for not having the abortion, and he said if I come after him financially, he will "fuck my life up."
Truthfully, I have zero interest in court drama, and I have no intention to go over him. 
However, here is where I need insight: I am terrified he will pop back in and try and take the child from me/try and get custody. 
Moms, what is the likelihood of this? Is this something I should be worried about? It's bothering me so badly, I can barely concentrate at work today.