Issues

So I am 24 and my dh is 32. We have been ttc for about five months now. I have been off the pill for five months and in those five months I have only had two periods. I know everyone is telling me that it could take a bit for my hormones to balance out after stopping the pill. But I'm worried that I won't be able to conceive. My periods have never been regular. When I was in high school I was lucky to have 5 periods a year! And my husbands siblings have all had issues conceiving, it took his sister 10 years of trying for there baby. So we think that he might have fertility issues too. We both want to have babies and are both afraid to go to a clinic to see if either of us have any issues.