I feel like he doesn't spend enough time with me

Ok so my husband works the craziest hours at work. This past week his shift has been 3pm to 5am(it changes all the time). My work schedule is 8am-5pm so we have barely seen each other at all this week. Our anniversary is 9/15 but he will be away for 4 days for a fishing trip during that so I was hoping this weekend we could do something special because we haven't really seen each other this week and we barley will at all next week. Yesterday I asked him what the plan for the weekend was and he tells me he has a golf tournament saturday and another one on Sunday. I really can't help but be so upset!! I just want to spend time with him. So this morning as he was getting ready to leave he asked why I was acting so miserable, so I expressed to him what I was feeling and that I just wished I was able to see him more this weekend. He immediately got mad and Said that if I wasn't happy I should leave because he is sick of me giving him a hard time every time he wants to enjoy himself and that he deserves it because he works so hard. Sometimes I feel like I am being selfish but on the other hand I know I'm not happy with just a hey how was your day ok kiss love you bye. I miss him!! What can I do??