Do you think that your desire to have kids overrules everything else?

I am married now, my husband hasn't had a a vasectomy, has fathered two kids previously, and I was super-paranoid shortly after we got married, that he had a vasectomy, and didn't really want kids, even though he told me he did want one more. After many months of attempts, my husband FINALLY produced a sperm sample, which showed that he hadn't had a vasectomy. But my fertility doc said that he DID have a low sperm count. I had an old boyfriend that I was obsessed with for years, but he never wanted kids, so I held out and waited for someone who did, but now I am in my late 30s, and scared that I waited too long.