Husband is mad.

So seems husband is mad because I refused to have sex this morning because I'm exhausted dealing with our toddler at night. Plus he hasn't apologized to me yesterday for saying I keep bitching when all I did was remind him he said he would be home at a certain time yet came home 4hrs later like usual tells me a time but comes home later. I got upset and told him at least I don't mind him going out unlike him made me paranoid that if I hang out with a friend they'll attempt to kidnap me or the kids. I told him its not fair I'm always home and he gets to go out. He never apologizes and now seems to be trying to make me guilty for saying no to sex. Starting to think we need therapy. Sorry ranting. I can't talk to anyone about these things.