Drinking and Smoking?!?!

So my husband and I have been TTC for about 3-4 months. We both drink (weekends mostly) and we both smoke (he smokes 1 pack a day, I'm more of the social smoker..a pack a week) Since we've been trying I have cut back a lot, and he has tried to quit, but is struggling with it. We know that they both are a factor of infertility but Could him being a heavy smoker and both of us drinking really be the thing that's preventing us from conceiving? We are both willing to quit for a baby!!! But we see SO many people that drink and smoke on the regular get pregnant :/