I don't want to quit! But should we??

My husband and I are pretty avid smokers! (Weed - just to be clear :) no tobacco. ) Everyday - usually 3-4x a day. We are TTC baby #1 and of course I've heard from my doc and online that we should stop smoking to help our chances, but I guess I wanna know from you guys...anyone continue to smoke as much as we do and not have issues conceiving?
We will quit if we get to the point where we feel it's necessary. But just wanna hear from you guys and see if smoking really makes a big difference in anyone getting pregnant!