Fertility?

Little bit of background my hubby and I have a 4 year old son, we tried for 5 years and finally got pregnant. I never seen a specialist. We started trying right away for another. I finally decided to see a specialist. My dh sperm count is normal, I'm ovulating normally and I had a hysterogram and everything is normal. No issues whatsoever. So why am I not pregnant? I have an appointment feb 6th but I thought I'd ask u ladies for opinions. Help me. I'm so confused!