Losing hope!!

Hello, everyone I am new to this site. My friend told me about it so I thought I would check it out. My husband and I have been trying for almost 4yrs now to have a baby. We have been going to a infertility doctor and everything. Back in April we became pregnant we were so excited and happy. But then we got bad news that it was a ectopic pregnancy. Our hearts were shattered and just felt so hopeless. They did catch it early enough that I didn't need surgery. After that we took a break and then started back up again the following month. But still no luck in conceving again. So now we aren't seeing the doctor anymore and just thought why don't we try on our own again. It's hard not to think about it or stress over it either. I really don't want to go back next year to the infertility doctor and go through all the meds and getting poked. If it comes down to that we would be doing the ivf. I am scared of that since it's different from getting the iui done. But I know I gotta pray and hope that god helps us have our miracle baby.