Refered to fertility doctor

I’ve just had my blood test done and had abnormal results i had vitamin d deffeciency and some hormonal imbalance. I was absolutely shock when i had the result and now my doctor is refering me to fertility doctor im so scared and dying inside i was crying and telling my hubby that maybe its my fault thats why we havent had any luck we’ve been trying for 3 years and it breaks my heart 😭