My husband is disinterested in having sex with me.

My husband is reluctant to have sex with me. We're 26 and have been married 4 years. We have a 4 year old daughter and between work and parenting we don't get a lot of time to ourselves. This only makes it worse when we do get the chance and instead of seizing the opportunity my husband would rather watch tv or play around on his phone or sleep. I've gain weight recently due to various circumstances of our life and I can't help but feel like his distance is because of it. He was never the "jump your bones" type to begin with but now if we have sex more than once a month it's a damn miracle. I know it probably isn't me since he has issues of his own such as depression, but it really makes me feel unwanted and ugly when I'm so interested and he is so disinterested. He flirts and tell me I'm beautiful and sexy all the time, usually at the most inopportune moments, but then later when we are alone there's just nothing. Eventually, when he finally needs attention then we have sex, it's usually at inopportune times as well but I never turn him down because it's simply too rare an occurence and I like having sex. (Or gasp) But even then it's obvious that it's entirely for his benefit rather than mutual. It's over in a flash and there is no round 2.... I usually end up sneaking off to take care of myself afterwards right after. The rare times when he is focused on both of us and he takes his time, it's much better. (I live for these moments)
I love my husband and I'm very very much still attracted to him, which is why his distance and rather one-sided performance are so hard to deal with. I've tried talking to him about it and he gets very defensive and accuses me of stereotyping him and thinking that all guys want sex constantly, etc. If I so much as suggest seeing a doctor to check for a possible medical reason he flies completely off the handle and usually makes me feel like a slut or an intensitive bitch for it. 
 I have no idea what's up with him. I know it's probably not my fault, but I can't help feeling like it somehow is....