When to ask for fertility referral?

Just a little back story. We’ve been TTC 3 almost 4 years now, we’re both young (24 & 25) and healthy! I started seeing my OB in July 2023 and found out I have a 2mm pituitary microadenoma (that has never affected my cycles and i ovulate on my own still)

I started Cabergoline for it Nov 2023 and then started my first round of clomid March 2024. I did 3 rounds of clomid and just finished my 2nd cycle of letrozole. All my other hormones are perfect and my husbands Seman analysis was perfect too.

I’m at the point where i’m feeling so defeated. I’m so tired of taking all the meds, doing the scans, and blood draws and just all these appointments for nothing to work. I’m almost to the point of giving up. We live in a town where there aren’t any fertility specialists and the closest one is 3 hours away. When is a good time to ask for the referral? I just don’t know when to call it quits with these meds for hyper ovulation. I’ve been looking a lot into IUI lately. Thoughts? Opinions?