Vitex pregnant

I just found out I'm pregnant and I was taking vitex for the last 3 weeks. Should I stop it cold turkey? I have an appt on Wednesday with the Doctor