extreme anxiety

i’m 20 and pregnant. i have a restraining order against the father so he cannot legally enter my life, he knows i’m having his baby. i really don’t think he cares though. i’m just really anxious that when the baby is born he will try and get partial custody, he’s extremely abusive, drinks a lot and smokes both weed and cigarettes constantly. he’s a shitty guy and i don’t want him anywhere around my child. it gives me panic attacks when i think about having to possibly drop off my child with him. i think if that were to happen i’d have a severe panic attack and not know what to do and that his family would say i’m crazy and file for emergency custody then i lose my baby for good. i’m freaking out i really am.