Please help!

I just got my bfp and this last cycle I have been taking vitex. It says not to take it when pregnant but I'm scared if I just stop I might misscarry? Does anyone know??