Drinking...worried!

Aspen
My husband and I started trying in December and we never got pregnant...we took this month "off" of trying because we had a trip to san Fransisco and we both wanted to have fun. Not gonna lie, I drank and smoked a little weed :/ I also drank and smoked a little on Saturday and Sunday because I figured if we weren't trying it would be okay...I don't have a lot of CM and bought preseed to use during sex because we had been using lube and I heard that affects things...well I was supposed to start today and usually I will get spotting the night before...nothing happened...then I took a test this morning...it's positive! I feel horrible and totally irresponsible because of the drinking and smoking this month! Has anyone else had that happen?! Should I be worried?!? Obviously I'm not going to drink or smoke now but have I already done damage? :(