My husband is backing out :(

My husband and I have been trying to conceive for 16 months. We are both younger ( 23 & 24) so I thought it would be fairly easy to get pregnant. At month 13 we decided to get checked to make sure everything was ok. I went through with the testing first, and everything was OK on my end. My husband has tried to give a semen sample 5 times and just cannot produce enough.. which may be the problem as to why we can't get pregnant. Now my Dr. is having him do alternative testing and set him up with a specialist and he is backing out... Saying "it will happen when it's supposed to.." I know he feels like he's the reason why we can't have a baby right now and I'm sure he feels guilty and I have tried to be the best supporter I can be for him but I can't help to be extremely down and depressed also, like he's giving up on the one thing we BOTH wanted more than anything. Just don't know where to go or what to do from here 😭 It's hard to just sit back and not try to figure out the problem that could POTENTIALLY be easily fixable..