TTC causing delayed ejaculation

My husband and I have been wanting to try to have a baby for about 2 years. I feel like it's been the topic of all of our conversations for that long. We kept waiting and waiting for the "perfect time"- which obviously never actually comes. But now things are going exceptionally well in our lives and we have more stability than we could hope for. So naturally we both agreed that it was time to start the baby making. Now all of the sudden my husband can't "finish". This past week was my fertile week and we had sex nearly everyday. He was only able to finish 2 or 3 of those times in the beginning of the week- and that was after 3 tries. This is so unusual for him and it's really worrying me. Obviously he has some reservations about trying to have a baby, but after 2 years of talking about it I am seriously confused. 
I am just wondering if anybody has been through anything like this at all? I'm worried about him and the hopes of our future family. And of course, I am left upset and confused as well.
Thanks so much y'all.