Marijuana and Fertility

I'll be honest here, my husband and I both smoke daily, we don't drink or use tobacco products so this really is our one vice.  We live in Los Angeles and do it legally with medical recommendations for different things, mine being Multiple Sclerosis.  I've heard it can affect sperm motility but does anyone know other possible effects on the man?  Or the woman?  I obviously plan to stop when Im pregnant but since I'm off all my other MS meds while TTC I don't feel like it's the right time now, when Im pregnant my disease will supposedly go into a natural remission.