Would you be mad?

So granted it has been 4 weeks since I had our son but my SO is basically telling me if I don't work out and lose the baby weight he's going to break up with me. I'm not fat by any means I just have a little extra pooch on my stomach. I weighed 99lbs before pregnancy. I work, come home take care of baby. And he doesn't. I honestly don't have time to work out and really just feel hurt because my image is obviously so important to him. In my head I'm screaming wtf I just had a baby I work and I'm tired all the time and you want me to get all dressed up and work out. 
Is this a normal thing for guys and I'm just over reacting and in my feelings?