Unprotected sex

I need advice. My bf and I started dating a while ago. I lost my virginity to him and I do not regret it. We are currently using the withdrawl method. We havent reqlly thought about birth control because of the following: he has low sperm count. It was so severe that he was not able to have kids for 5 years while he was with his ex. I have an appointment in december to get more info about birth control but what do you guys think? Should i be worried??