When is it time to see an actual fertility doctor?

I’ve been officially trying for a year now. I stopped my pills last December and had my first withdrawal bleed during the first week of January. During this time, I had no idea that getting pregnant would be difficult so we were just winging it; I didn’t track or anything. Then at the end of January, I got what I assumed was my first natural period which lasted 7 days. Then when I was a week late in the beginning of March, I tested and got a negative. I continued to test every week that month and still got negatives. I finally went to the gyn for a pap in April and she told me to let her know if I hadn’t gotten my period by May because she would prescribe me provera. I didn’t get my period, so I took provera and got a 5 day withdrawal bleed 5 days after taking my 10 day prescription. Once again a month later I was late but I was still getting negatives. So she gave me another prescription of provera in July and then had my blood work done on CD3. She also ordered an ultrasound that showed my lining was thin with scarring. In August she gave me more provera, and then I had an HSG in September in which the PA who did it told me that everything looked fine. So after all of this, my gyn told me that my bloodwork suggests I have pcos and that I needed a hysteroscopy to remove scar tissue from my previous pregnancy. She also put me on metformin (now on 2000mg). I had a hysteroscopy during the first week of November and she was able to remove the scar tissue. When I saw her for my follow up 2 weeks later, she gave me more provera then started me on 100mg of clomid CD5-9. I began OPKs starting CD12 all the way up until CD21 and never got anything other than a flashing smiley face. My CD21 bloodwork showed that I hadn’t ovulated. So my gyn told me to up my dose of clomid to 150mg then come in for bloodwork again on CD10 and 21. So right now I’m on provera, currently waiting for my cycle so that I can start all over again (this time I won’t be using digital OPKs).

So my question is, when should I see a fertility specialist? I’m really frustrated because we didn’t even try with our 6 year old and although we’ve always wanted more, we tried to do “the right thing” by making sure we were older and married with stable careers and now we’re dealing with this. And honestly I’m not even sure that I could handle <a href="https://glowing.com/glow-fertility-program">IUI</a> or <a href="https://glowing.com/glow-fertility-program">IVF</a> emotionally or financially right now.