Frustrated

Does anyone else get frustrated when their baby throws a “tantrum”? I don’t mean to but his scream really hurts my ears and it’s over stuff like I won’t let him play with something he shouldn’t play with. I know it’s normal but not sure how to handle it or handle me getting frustrated really because I don’t want to bug sometimes his scream is so high pitched I can’t help but yell “STOP” it’s just a reflex. I feel awful about it, I guess I need to work on my anger?