So damn frustrated!!

My husband is in his 30’s and I’m in my mid 20’s. I just want to have a family soooo bad! We’ve talked about it and he said that we can stop using protection and just let it happen but now he never wants to have sex! He says he’s not ready but I can’t help but feel like he’ll never be ready 😔 also, I have PCOS and my doctor said I’d need to take medication to make me ovulate and he said he doesn’t want me to take it and that it makes him to nervous and maybe in 5-7 years I can take the medication and we can actually start trying or at least re visit the topic. I don’t know what else to do. I’m at the point where I’m about to get a damn sperm donor! I don’t want to be an older parent.