Infertility odds?

So... I'm still waiting. I've always been regular but now I'm 21 days past af's expected arrival. Only BFNs. No symptoms of pregnancy except the missing period (unless blowing up on the hubby for seemingly not much of a reason is one!)... Went to the doc and waiting for results of the blood test to find out if this is pregnancy or other random problem.  I had a blighted ovum 10 years ago (different spouse). Husband and I have been trying for two years. Then my BIL tells hubs that he just found out that he's sterile. So... Now I'm freaking out while I wait for the docs results. My older brothers don't have kids, my little brother has fertility issues, my brother in law is sterile... Seems like the odds are against us. Or am I just freaking out for no reason?