should I just let it go?

So my hubby asked me graciously if his buddy could come over tonight and play cards. I was fine with it so long as he helped me tidy the house a bit before. So I did a lot of tidying while he talked to his friend on the phone but then I had to feed the baby and get her to bed and there were still things that needed done and he was still on the phone. So then my baby starts screaming and fighting sleep and I'm stressed to the max because the house is still a mess. Finally I get baby in bed his friend isn't over just yet. He gets off the phone and I say you didn't help me at all! He then told me "I picked up the bathroom. Why do you always bitch and complain that's all you do!" Now granted I was very frazzled because my baby was stressing me out fighting me so hard earlier so I could've been nicer. But when he said all I did was bitch I just froze. He's never sworn at me once in our entire marriage. (5 years) I sighed and ignored it and moved our baby's playpen when her toys started all singing at once and the two of us shared a laugh trough the tension. But should I try to tell him I'm sorry or express how much he hurt me when he said that? He's not the type to apologize ever so I'm not expecting him to say sorry in return. What do I do? They're playing cards now and I'm in my room. Should I just let it go?