feeling hopeless

2 cycles of clomid .. 1st was 50mg and got one follicle 18mm 14cd.. next increased dose to 100mg and no mature follicle ( 3tiny ones only) on 14cd. changed my medicine to letrozole this cycle and just 15mm.. doc said not very hopeful about this cycle also.. said have to start thinking about going to the specialist... feeling so hopeless .. going to be 3years of marriage this month end.. 3years trying and nothing even with medication.. feeling so hopeless and depressed.. all around I look I just see people disappointed in me... my parents , in laws everyone.. saving grace is my hubby still loves me.. I actually stated crying at the doc office ... feeling that hopeless... 
will it help going to specialist? 
Have done all other test and they are fine.. it's since last cycle that they told infertility due to anovulation