Should I let him keep visiting?

Ok so I've been letting my ex visit our 7 month old whenever he wants, but today he took my phone and started going through so I tried to take it back, but he wouldn't let me. I pushed him and tried to get it, then he pushed my against the wall and choked me in front of the baby and his 4 year old. After that he pushed me on the floor and grabbed my hands and arms really hard. I'm pretty sure I'm going to have a few bruises. I told him to leave and never talk to me again. Is it wrong of me to not let him see his son anymore?