20 & ttc?

Okay I just turned 20 my DH just turned 24 and we have been together for 4 years. I have the worst case of baby fever. I'm dying to have a baby! I know hubby and I are responsible enough and can emotionally and financially support a baby. But he doesn't tell me how he feels about it. He knows I want a baby. He doesn't try not to get pregnant. In the past 2 months I can think of 3 times he has pulled out. I'm not on BC and he doesn't use condoms. We have never really been good about using protection but I've still never gotten pregnant. What are the chances of one of us being sterile? What are your opinions on ttc at 20?