Please help TTC Baby #1

Me and my husband have been TTC baby #1 for almost 3 years(in october). We both have been checked out by our doctors and are healthy. I recently ask my OB if I could get a preauth so me and my husband could start seeing a fertility doctor and my OB refused to sign one he says I'm to young to see a fertility doctor (I'm 21 and hubby is 29). This would be my first child. Things are just real tough with what it seems like EVERYONE around me getting pregnant. Please help been real down about it.