Weed IVF Fertility TTC

I’m really bugging out !!! I hadn’t smoked for months because my husband and I were ttc and going through Iui and fertility treatment . After my first egg retrieval that ended in 1 healthy embryo doctor told me to do another with a month off i smoked very small amounts 1-2 x a week for about 2-3 weeks .. I’m panicking I’m ruining my chances egg quality and what not .. my husband doesn’t smoke at all so I am carrying all this guilt ..last night I toked a Friends joint 3x after stopping this week!! What’s wrong with me ?! I want this so bad but I cave at temptation... anything to help reassure me.. my head is spinning and it’s 5am here!!!!