So Frustrated and HURT!! Why wont he touch me?!

I have been with my fiancé for 3 years now. Right from the beginning I noticed a lack of drive for intimacy. I was honest about it with him and he said "its a confidence thing". He assured me it would get better. It hasnt got any better. We have times when it gets a little better but it seems like every day life stuff gets in the way. We have had some pretty traumatic things happen like his Dad passing away suddenly. Of course I wasn't expecting him to be intimate during that time. I am really hurting over all of this. I feel like he doesn't desire me. I feel like my confidence is getting lowered because of all of this. I feel like it seems like such a chore for him to be intimate. Now we are trying to conceive and have been for the last 13 months. We have been all checked out and nothing is wrong with either of us so we work. I know its because we arent intimate enough and there is a lack of connection. I have no idea what to do. I am not happy and this causes a lot of fights between us. I am scared I wont ever have a baby at this rate. I guess I just need some advice or help. I am just so hurt!!!