He keeps breaking up with me..

My partner and I have been together for 5 years and have a 2 year old. We split up for a few months after about a year. We split up again from August last year until November after a stupid drunken argument and now he's broke up with me again? And again over a stupid argument. He always comes back eventually but I'm sick to death of being broke up with over nothing and then once I start to feel myself again after a few months he comes back. I love him and don't know what to do. I've really lost my sex drive lately (I just feel ugly after having a baby, I've gained a lot of weight and I'm just not feeling it anymore..but I still try to have sex with him because I don't want him to go without) and we came back from a party on Friday night, I said I had a headache and he started saying how he is sick of never having sex and how I must think I'm that gorgeous that I can with hold it from him. He said I'm not "even that nice looking." I started to cry and he told me we were over. Mind you I hadn't even said a word other than sharing that I had a headache and he went off. I slept downstairs and he kept texting me to come to bed but I felt utterly disgusted. We don't live together (though we have just sent an application for somewhere of our own before this) so I went home and he texted saying sorry but suddenly wants nothing to do with me now and I just wish it all wasn't so childish. I want to sit and talk it out like adults but he just can't seem to have an adult conversation. I don't know what to do and some advice from someone looking in might help! thank you!