Am I neglecting my husband?

I'm three weeks postpartum. Every day since we've been home from the hospital, my husband has been not-so-subtle about how horny he is. He wants any sort of sexual contact. But I am just not feeling it. I am so overwhelmed with my son that I have no energy or motivation to perform. And quite frankly, the baby blues are hittingme real hard. Well, last night while my son was taking a nap, we started making out and fooling around on the couch. Of course, my son decides to wake up before it gets too far so I get up and attend to him. And my husband gets upset. I brush it off bc wtf am I supposed to do? Again this morning, my son was taking a nap so my husband and I slip into the shower to fool around. Of course, my son decides to wake up again before things go too far. He starts screaming and I go to get out of the shower and my husband throws a fit. I tell him that I can't fool around while my baby is screaming. He doesn't understand my reasoning and continues to throw a fit and now he is pouting in the bedroom. I have no idea what to do. I want to please him but this whole "becoming a mom" thing is exhausting and consuming my entire being. I know it's only been three weeks and that it will get better but my husband fails to see that. Uuuuuugh.