Wine drinkers

Hubby and I are trying to have a baby and this is our first month actively trying. We had a special weekend and everything. I drank quite a bit of wine. Should I stop drinking wine now just in case or is it so early that it matters???