Custody/Visitation advice!!!!

My ex fiancé and I split up a couple months ago. This was due to him being very verbally abusive, him stating multiple times he wanted to kill himself and not getting help for himself, and him screaming and cussing and slamming doors in front of our infant. He has only fed her maybe one or two times and that’s with me making the bottle, has changed only a couple diapers, and when she cries, he hands her back. Right now we have no court order and we tried to keep the courts out if it so he has visitations at my house 3 times a week. I took my daughter to his moms house yesterday to see her and his sister and they verbally attacked me the moment I walked in. Then unexpectedly, my ex walked in cussing and screaming in front of our daughter telling me to “get my ass outside we need to talk” and I said absolutely not? And he said “well I’ll see your ass In court.” Now keep in mind, I let him take her one time for a couple hours by himself to his moms house for his grandmas birthday and she screamed the whole time while being there (she always does, she doesn’t like being there or doesnt let anyone over there hold her because they’re loud people) and just because she was crying they gave my 5 month old Tylenol without taking her temper rather or calling me and consulting with me about it. Nor did they call me to let me know my daughter wouldn’t stop crying, Therefore they cannot be trusted with my daughter obviously. And I was verbally attacked by everyone yesterday so I have decided to only let them see her in public like at a park or at a restaurant because I will not be verbally attacked every time being there. Am I wrong for not letting my ex take our daughter or his family to watch her alone?