drinking & smoking before TTC

So odd question, but has anyone smoked a lot of marijuana in the past and still been successful TTC? My husband used to smoke a lot back in college, he doesn't anymore, but I wonder if that would still have any affect on his sperm? I know it can affect it while you smoke but didn't know if it would have caused "permanent damage"