Keeping a secret

I have been off of birth control for over two years now. I am 27. My husband and I have "pulled out" off and on for those two years. We have been actively planning and trying for baby #1 for about 4 months now. I'm still not pregnant. I have gone to the doctor and told him my concerns about it, he drew blood to test if I am ovulating (I never have pains) and to see if I am fertile. My husband doesn't know that I told the doctor. He knows I'm worried, but he thinks it is ridiculous. Should I not be worried? Is my husband right? I get my test results tomorrow, hopefully.