He’s becoming violent towards baby

My sons 9 months old

He’s been waking up every 2-3 hours every night since he was 5 months old and usually goes back to bed once we give him milk, CIO and other method haven’t worked

The baby wouldn’t stop crying and we both tried everything, it was 3am he got louder and louder and he yelled at the baby and smacked the crib I thought he was going to hit him.. I yelled at him and said things I shouldn’t have but he’s never like that he’s always calm and this is the first time he’s done that

I am afraid he would hit the baby if I wasn’t there.. I just don’t know

I’ve never been worried the baby would cry too much around him he’s a good dad