Is God's plan for us not to have children?

Je
My husband and I have been trying for 4 1/2 years. We haven't seen a specialist but my regular Ob/Gyn has done all the tests she can. All tests come back normal. I'm at my wits end. It's hard for me to watch friends and family pop out kids any time they want, or even "by accident." 
My question is ...
If it's God plan for us not to have children, would He have put such a deep desire in my heart? That just seems cruel and I know God is too loving for that.