Sex in marriage??

Hi I busy need some advice, am I wrong in thinking that there are issues in my marriage because my husband just does not want sex at all 😔 we've been married 2 years, I've been off contraception for 3 years. I really want to start a family and we've had arguments in the past because I was feeling that he wasn't that bothered when I suggested we start trying. Since coming off the pill our sex life has been next to none existent. We went away on holiday last month and I made such an effort to spend quality time together and get dressed up for nights out. The first night I had a few drinks to give me the courage to make the first move, I tried to make it enjoyable for him but it didn't happen he said he'd had too much to drink. I felt devastated but I didn't want to cause an argument so acted like it was ok. A few nights later I tried again and we had full sex, it wasn't amazing but I thought it was a step in the right direction. Unsurprisingly things have stopped again some coming home and tonight I just feel completely useless. I went to the gym (trying to make an effort to lose weight to look better naked) I went to start making him a nice meal and his response 'your not starting that are you I'm going out in an hour do it when I get back' I just seriously don't know what to do, I find it so hard to talk to him about how I'm feeling and I just keep thinking what if this continues for the next 2-3years and it gets to the point where it's too late and il never get the baby I desperately want 😞😞