Scared to admit I have fertility problems

Hi. I'm new to this, so please bear with me. I'm not sure who to talk to. My husband and I have been married for 7 years, together for 13.  We have tried for over 2-3 years to get pregnant and no luck. I have been to the doctor and so has he. They all say just to keep trying and chart everything down. My biggest pet peeve is when we are around family and they all say" wow, I'm surprised you don't have kids yet."  I'm just very stressed about this.  Thanks for letting me vent.. 
Jessica Perez