Can't stop arguing about sex

So my boyfriend and I have been together for almost 2 years and have been living together for 1 year now. He is a very sexual person. I am someone who is only in the mood if conditions are just right. If I've had a long day, or I'm just tired or stressed in general, I don't want to have sex. I am in school and work full time so this is the case more often than not. I think that sex 2-3 times a week is normal, but he thinks it should be more like 4-5 times a week. We have been arguing about sex a lot recently because he feels like I am not making enough time for him and (even though he hasn't said it out loud) that I am being selfish. I want to fix it, but the more we argue about how often we have sex, the less I want to have sex with him. So then I never come on to him, and he gets even more upset. It's so draining to talk about and it's just taken all the fun away from sex. Has anyone else had this problem before? If so, how did you handle it?