Please I Need Some Advice

Me and my husband have been TTC for a couple of months , but there's one problem . My husband used to be a heavy cocaine user and I've heard that reduces your fertility . Is this true ? We are planning on going to a doctor this month , since we've had no luck . I'm just really scared this is going to be a problem for us.