How do you deal with a boyfriend that

Oversteps his boundaries when it comes to your child.. I have a toddler and I’ve been with my boyfriend for 6 months.

My child throws tantrums like any other kid and I’ll catch my boyfriend like telling him he’s goin to take his toys and all these diff punishments. (They’re non violent but I’ve still asked him to let me handle the discipline as it’s not his child not his job.) today we were out to eat and I went to get more food when I came back my son was crying and I asked why. He told me my boyfriend broke his toy. So I told him that I’ve asked him to stop messing with my son plenty of times. He goes “oh we were just playing around and I accidentally broke it” I called him on his Bull shit and told him I know he was trying to take it from my son. So then he admitted that my son hit him so he was trying to take his toy and “throw it away” as a form of punishment.. am I right to get annoyed and upset? I keep telling him to leave my child alone and let me handle discipline but he keeps over stepping his boundaries. I’m considering breaking it off but I’d hate to since he’s such a good guy