Boyfriend masturbating...

I know this topic comes up SO much. I’m usually with my bf 4/7 days of the week. We usually do it every 2-3 days, but of course there are those weeks where we don’t do it for longer. I know he masturbates. I found out maybe a year into our relationship. For some reason I was upset and he told me he wouldn’t do it anymore. 3 months later I found out he never stopped, but I realized it was okay because it didn’t effect our sex life at all. Fast forward 3 years later, I still feel the same for some reason. I’ve been trying to not care about it, but I do. I actually left last night to go home. Before that I was at his house hoping to have sex, but he was just not in the mood. The next day I come over to find out he masturbated after I had left (not like right after, but a few hours later when I was asleep). I was upset because he knew how much I wanted to do it that night, but once I left he did it to himself. He told me he wasn’t in the mood when I was there and has always told me they are two different things (masturbating and sex). He told me we could do it today (maybe even twice). But since I was all upset today, he’s annoyed and that definitely doesn’t put him in the mood. Is this true...that they’re two different things? That him masturbating doesn’t effect our sex life? If anything he masturbates once every 2 weeks. Should I even be concerned?