Hubby is too tired for sex..

I will try to make this as short as possible. I have always had a little trouble getting pregnant, it always take a about 3 years. With my daughter I was 17 when I got pregnant and for 2 years we had unprotected sex (he was a terrible person tho) I had her at 18 and then when she was 6 months got pregnant with my son, had him at 19. I then left their father (he went to prison smh) and met my husband. I was on the pill and he was fine with that as he is older than me and had 4 of his own children. I had problems with birth control I bled for 4 months non stop. I went to the hospital and they told me I couldn't have anymore kids. I was distraught. I was only 20 and I met the man I knew I wanted to marry. I got off BC and we had sex constantly I never ended up pregnant. After 3 years we decided to get married I had come to terms that I wouldn't have anymore children and should be happy with the family I had. After 3 months of being married I was pregnant I couldn't believe it. I had our daughter that same year. I am now 29 and have expressed to my husband that I want one more baby. 5 years have gone by since we had our daughter and we were having sex constantly, here recently since I downloaded this app and have really been TTC I quit smoking, I don't drink, I exercise, I'm dieting in a healthy way.. suddenly he is too tired for sex. I am a little over weight and I have self esteem issues to which my mind goes nuts and I end up telling myself terrible things about myself. we've talked constantly about this and he swears it's just that he's really tired, but I get insecure, my question is, am I maybe pushing too hard for a baby? Or am I too into my insecurities? Thanks in advance for any responses.