Masturbation..?

My sex drives has been CRAZY lately! But my husband and I work completely different shifts. We have quickies as we see each other for 1-2 hrs but even after that I still want more. We're trying to conceive too. Will me masturbating have any affect to lessen our chances of conceiving..? Bc I've heard that if a guy masturbates too much it can lessen the sperm count.. Any how, someone please answer..? Thank in advance