what should I feel about Doctors opinions?

I am 23 and TTC I have been married for a year now but have not been preventing from getting pregnant for about 3+ years I am so to make an appointment to see a fertility doctors but I feel he might say I'm too young? Should that bother me?