Would you feel guilty?

I work 3-11pm, never fall asleep till 1-2am. My BF gets home from work at 6:30am and always wants sex but I'm so tired from working and not getting sleep, I turn him down. Now if I didn't work the next day, I wouldn't. We have two kids, 2&3 year old so it doesn't leave much time for sex. I leave for work when he is just getting up. I feel horrible cause I hear these other women have sex every day and I turn it down and if we're lucky, we have sex once a week.