Leaving my gyno

So at what point do you give up on your current ob and go elsewhere? I've been going to a certain dr since November, it's actually their nurse practitioner. I've had so much trouble getting ahold of anyone to make appointments, they want to shove clomid down my throat. I told them I want to redo the 21 day draw (this will be 3rd round of the draws), because we figured out I have a longer than 28 day cycle so drawing on day 21 is not correct. So I was asked to call when my period started so the nurse could figure out days to do draws and send lab orders to the local hospital. Well my period started 6 days later them expected on the weekend so I called Monday to call and I got a message stating that they were closed for a week. So no big deal, I could leave a message. Phone goes to busy tone so I can't. Have tried to call back with same result. Plus the hospital this dr works for I've heard awful things of how they take care of their new mothers. So I really don't want to continue with this ob. There is another hospital the same distance from where I live as the current hospital that is much better. I know switching doctors will be a huge mess and God knows when they will get my infertility issues figured out but I think it's worth it since this current doctor and his office seems to just give people the run around. Should I really leave? And if so how do I go about getting all these records moved around with as little headache as possible?