SO semen harmful when pregnant

I know that the zika virus can be found in a mans semen. But I'm also highly concerned if certain drugs or pills are found in mans semen as well. Like the anxiety pill alprazolam? It's harmful if pregnant women take it but what about her spouse? What about marijuana? I know only a portion makes it to semen just like how a little bit of cocaine can as well. But I'm more concerned about the certain pills, alcohol and marijuana intake my spouse takes and does. I'm just concerned if it can harm our baby with what he's putting in his body. Is anyone else concerned or thought about this? I just don't want to sleep with him with him ejaculating in me each and everytime when these things are possibly in his semen effecting my child. I just want to know if it's safe to continue our regular sex activities with him being able to climax without our child being harmed. Help!!! Opinions!