At what point should I find a new fertility specialist?

I'm getting fed up with my fertility specialist. I want to find a new one, but I also don't want to have to start over with a new one. I'll try to keep this short.

My husband and I have been trying to conceive for almost 4 years now and we started seeing a fertility specialist in the spring of 2019.

I started out with a pelvic exam and tons of lab work. Found out I have PCOS and hypothyroidism. I was on birth control for a few months to lower my testosterone, and then started letrozole. I have done 5 rounds of letrozole (ovulated on rounds 1-4, no pregnancy, did not ovulate round 5). So for the past year, they have been working on me, but they still have not ordered a sperm analysis. I was told they would hold off on that because male factor infertility is not as common as female factor, and we already know what's "wrong" with me. I get that, but male factor still happens. There's a chance there could be something wrong with both of us. Why not have him tasted so we can rule that out? What's the point of fixing me up and trying all these things to get pregnant if he could be shooting blanks? I was told this in August 2019 and that was kind of the beginning of me wanting to switch doctors.

On top of that, my first appointment was with the nurse practitioner and my second appointment was with the Dr, so I go through both of them equally. I'll be messaging one of them through the portal and they switch back and forth responding to me. So within one conversation, I'm talking to two different people and they often contradict each other.

Some new information has been brought to my attention this week. So I was prescribed a medication for my thyroid spring 2019. I have hypothyroidism, which means my TSH level is too high and my thyroid is underactive. In August, my Dr wanted me to have my testosterone retested before my next appointment (so I could stop the birth control and move on to letrozole) and I asked her if I should have my TSH retested as well. It has not been tested since I started the medication in the spring and I wanted to make sure it was working correctly for me. She went ahead and put in the order for a TSH lab. If I hadn't asked for it, who knows when they would have tested it again. So at my next appointment in August, my Dr told me my levels came back good and I have continued the medication for the past year.

I'm almost out of the medication and do not have any refills left, so I put in a request to renew it over this past weekend. The next day I get a message from one of the nurses saying in my August lab result, my TSH was too low, meaning I was on too high of a dosage and my level fell into the hyperthyroid range (meaning my thyroid was now overactive). She asked me if the Dr had changed my dosage after that because she couldn't find a note stating it had been. I told her I was never even told my TSH was low. I told her my current dose and that I couldn't find what my original dose was.

So essentially, this past year has been a waste. All the time and money we have put into trying to get pregnant was all for nothing.

I'm feeling so defeated. I'm torn between "I can't do this anymore", and "I can't give up".

EDIT:

I thought I should also add, at my August appointment, after the nurse came in and asked her questions, took my blood pressure, etc, my husband and I sat in that exam room for almost 3 hours before the Dr finally came in. Between the time the nurse did her routine things and the time the Dr walked in, no one else came in. So in the 3 hours we were sitting there, not one person came in to check on us, let us know there was going to be a wait, nothing.