Feeling down today...

Ever since we've been TTC I have really tried to not make our sex life all about making a baby. Well this past week has been fertile week and I ovulated Tuesday. So I've been trying to have sex everyday day this week. We'll last night (night after O), I tried and he didn't seem into it at all, so I asked him what was wrong and he said that his penis hurt from having to much sex. So I was just like okay well we can just do it in the morning (I still really wanted to do it though). But he insisted that we do, but I told him no it hurts well just do it in the morning. Well now it's morning and we did have sex but only after I had made a comment in the shower. He could tell I was thinking about something and asked what was wrong. I didn't want to say anything but I did and said "how are we ever supposed to get pregnant if we never have sex?" I know I shouldn't feel this way but I do. How do you guys get over this feeling? I'm going nuts. I feel like if we don't do everything perfect that we'll never get pregnant! I'm losing my mind......