I need opinions on if I should switch Dr's or stop trying

My third cycle on clomid and it's day 21. Progesterone 4.9 with mid luteal phase estimating 5.16. Getting a second progesterone in 2 more dsys. Dr. Said if not successful this month will go to 300mg clomid. I could barely take side effects at 100mg. I want letrozole but he won't give until my 5th cycle after I try clomid 300mg. Should I see another dr. Or just give up or see if I can just talk myself through 300mg clomid? What would you do? I'm fighting back tears I'm so upset that I can't just keep higher levels.