HELP-coparenting turning custody battle

I had essentially an open door for my son’s dad to see him whenever he was able. I have gone to some of his family’s homes so they could meet our son (who is only 2 months) and have been nothing but accommodating. We came to an agreement on a schedule a couple days ago. The next day he asked if I was going to get off of state aid to depend on his own setting up child support. In short I said no that I don’t feel comfortable putting both my boys needs on the line for nothing but a financial benefit of his. He started threatening a custody request (which is slightly silly given how agreeable I’ve been and that he gets more time/opportunity for time before this) and then brought my apartment into it, which is my parent’s detached apartment, saying it’s not permitted or whatever. My dad called him, I don’t know how this call STARTED but I know within the call my son’s dad said “you shouldn’t have let me fuck you daughter” and his sister (who I have messages from saying he’s a drunk) said “he’s so worried about his safety because he’s in a non permitted apartment”.

All because 1) the state is mandating child support unless I get off aid and agree to something with him privately but not sure if that’s even a valid option since we already have court for it since he disputed it.

And 2) I expressed valid concerns about him having him alone in his home because of his drinking problem, lack of necessities for our son, and his roommate who has lost his own child due to drug abuse.

I have much of this in writing. I just don’t know if there would ever be a scenario where they take our son from me? I’ve lived with my parents essentially since I had my first son (3 now-not his) and have been in school/working towards our future since before/after that. I’ve always worked little jobs to help pay for things, and have had to be on state aid for 2ish years. Especially now with the two I have no idea how I’m going to pay for childcare and work I’d be working to pay for childcare so I’ve stayed in aid, the goal was to get to my older son being preschool age and then I could work but now with my second son its Tricky.

Also not sure if I should continue visitation at this point, he’s been extremely inconsistent and now is not welcome in our home because of all of this (which in caregivers home is best for a newborn esp close to bed time is what I’ve read) and I’m afraid he may exercise the legal right to just not return him since we don’t have a custody order in place. I don’t trust him. Period. And I’m afraid he will do something incredibly dumb. I’d even be reluctant to do visits with me there because of the threats that have been made