Who’s in the wrong, him or me

So the guy I’m talking to has this habit of taking things to heart when it’s not it’s only a joke and subsequently hanging up the phone. When I say something as innocent banter or to purposely take the piss, he hangs up the call and then I have to call and apologise. But when he says something to me that’s of the same context as what he gets mad at me for I don’t get mad cause I know it’s a joke or if I do, I hang up but he doesn’t call me back or ask if I’m ok like what I do to him. For example, today he told me to fuck off and told me I was annoying. Hearing that kind of hurt, but I didn’t hang up I laughed and said “fuck you too then” so he hung up the call then I had to call him back. I feel like the man of the relationship. I’m always babying him and calling him. I don’t feel like I get attention. Am I wrong for being mad? Is this red flag for controlling and manipulative man? Pls help.