I’m afraid my bf will hurt our son

I asked my bf/babies father to watch our son for a few minutes so I could finish my college assignment, and he can’t even go 5 minutes. He starts yelling at him to stfu and that when he’s older he’s gonna learn real quick that he’s not gonna put up with this and that our son will hate him and stuff like that. And how he wanted to bust his butt. Then said fine you wanna cry for no reason you can sit on the floor and let’s him fall over and he starts crying and he goes see that’s what happens when you can’t keep your balance. Then he puts him in the swing and said you can sit here until you stop crying. My son looked up at me with these sad eyes wanting me to save him, so I started to get up and my bf goes sit tf back down he’s like because you baby him too much. My bf then yelled at him to stfu and go to sleep and that he was getting on his nerves and was crying for no reason. I told him he was baby and all that is uncalled for and he was like yeah cause you baby him and he’s gonna be a spoiled brat. All because he didn’t want to take time away from his video game to give our son some attention. Like who treats a 7 month old this way?? This is part of the reason I’m afraid to leave because if I’m here I can see what’s going on and protect him. If we split up, I would be a nervous wreck the whole time he was gone and would fear my son coming home beat up. My bf underestimates his strength and when he’s mad it takes him awhile to cool down and I don’t want him taking it out on our son and not realize what he’s doing until it’s too late. I feel so bad that this is the kind of father that my son has. Our courthouse judge is very adamant about giving the father visitation. I’m all for letting my son know his father, if I didn’t fear for the safety of my son. This kind of behavior will impact my son for the rest of his life if it continues this way. I want the best thing for my son, but idk what that is right now. I know I need to leave, but I fear that if im not here to keep our son calm and that my bf was here alone with him not knowing what to do that he may harm him. I need some advice please...

Edit*

He has also made references about throwing him at a wall and when I say something he says he’s just joking, but I don’t see how that’s funny.