My heart hurts. 😞 Things are getting physical.

My bf and I have been together for 5 years. We have a 2 month old daughter. I have the issue of not talking to him because Im afraid he'll get mad. And he does, but I've been working on it and voicing my opinion more lately. He does not like it when I hit him with that truth! He does what he wants, with who he wants, when he wants for how long he wants. I have to call him to come home to us and by the time he gets home Im aggravated and don't want to spend time with him anymore. Its a screwed up situation. We sit in silence and have nothing to say to each other. I used to get emotional and cry about it. Now Im just fed up and take him as a joke because this "man" would rather run the roads than spend time with his family. I guess I laughed when he was staring at me like I disgusted him and he jumped up grabbed me by my face and pushed me to the floor. I've always told myself that if anyone were to get physical with me I'd leave. Not wait for it to get worse. Hes never done anything like that before. I can't fight him back Hes twice my size. He threatened to take my daughter and I can't physically stop that either unless I take legal action. How can he hurt me and play the victim and threaten to leave? HE did this! Hes a narcissist. What's even more stupid is that he's the one who hurt me and hes the only one who can make me feel better. I just want to be held by the person I love who is the person who hurt me.. My life is a mess. Idk what to do.