How worried should I be about my boyfriends history of drug abuse?

Before I met him, my boyfriend was kind of going through a thing with drugs. He won't tell me specifically what drugs he's done he just says he's "tried everything." I've heard that this can have an effect on his sperm and I was just wondering how worried I should be about my growing little baby? The drug use isn't super recent or ongoing or anything. I'm very scared.