Am I being ridiculous? Some TMI

So I found out that hubby has a low count. And I've read from multiple sources that when ttc you should (among other things to increase sperm count) only BD once every 3-4 days. Well last night we BD, and he couldnt cum, and he told me that he pleased himself that morning. Now I usually don't care about him materbating. But I can't help but be upset that he did. :/ my fertile window starts on Saturday. So should I even really be upset?