Vaping/Smoking Weed TTC

Me and my husband are preparing to start TTC our first little one. Both me and him vape/smoke weed on the weekends because that’s our leisure time. Would that have any impact on our chances of conceiving? I’m 25 and he’s 27. Other than vaping/smoking weed we have a pretty healthy/active lifestyle