3 months until refills... Husband says don't re-up.

Hmmm.... where to start! Well, my husband and I both want to have a child. We have been married a little over 2 years and have been together overall for 5. I was adamant about not having children ubtil I could get everything in the universe figured out first. Haha! But seriously, I have been diagnosed with endometriosis and have lost a child once when I was 19 (I'm 24). He knows about this and is concerned that this may have caused more scarring. I have met with doctors and I have discussed the hell out of our situation with professionals, but I want to ask you ladies what you think. I have 3 refills of birth control left before having to call my doc to call in a refill. My husband and my doc both said that maybe it's time I stop taking it and see how things pan out. We aren't struggling at all and have wanted kids so badly, but I have been so terrified of the idea. I understand that there honestly isn't a "perfect time" and I've come to terms with the fact that I'm not Superwoman, lol but I still can't shake the anxiety. But I want this as much as my husband. Maybe I'm rambling. Any thoughts?