Doctor basically told me to stop trying and I'll get pregnant.

My husband and I have been TTC #1 for almost four years. I got diagnosed two years ago with a pituitary tumor (micro adenoma) that causes hyperprolactinemia. Took medication for that alone to get my prolactin levels in check and we got pregnant 3 times, all ending in miscarriage. We then tried letrozole (I didn't react at all to clomid), fsh shots, ovidrel and progesterone. I didn't get pregnant with their "fail safe" drugs. We stopped fertility drugs over the winter and I even got a faint positive in February but again, I lost it. My reproductive endocrinologist wants to go <a href="">IUI</a>/IVF route - but I don't want to dish out all the money before knowing why I'm miscarrying. 
Went to a second doctor to get another opinion and she basically said if I stop trying to get pregnant, I will get pregnant. Seriously?! I can't believe that level of ignorance is even allowed in an OBGYN office.