Need permission to masturbate? (Read description)

My bf and I have kind of had this thing where we don't masturbate because we're afraid it will reduce our sex drives, thus reducing our frequency of having sex. And if we really really want to, we kind of let the other know beforehand.

I'm not sure if I want to do away with this rule because sometimes I just want to masturbate without informing. Yet at the same we only see each other 3-4 times a week now, and I'm a pretty sexual person...

Please note before judging: I love sex and when we would masturbate freely in the beginning of our relationship, we definitely noticed a decline in our sex drive that night. We both know that when we do NOT masturbate separately, then our sex is better because we're hornier. If I masturbate in the morning, it takes me forever to finish from sex at night, same with him. We also used to live with each other so we were always available to please each other, which was kind of the idea. Now we don't, so it's not as clear as to whether we should keep this unspoken "rule" up...

Thoughts?