Vitex when pregnant

Hi, after one year of TTC we got our BFP! I’m convinced it’s because I was taking Agnus Castus (vitex in the US). I’m just over a month pregnant and I’ve just noticed on the package that you shouldn’t take when pregnant. I don’t know if I have a progesterone problem, my Leuteal phase was 10 days for my past three cycles and I’m convinced we conceived due to this Agnus Castus supplement. My husband says I should stop taking as it might harm the baby, but I’m worried it will end in MC, but also worried vitex will cause harm and also may result in MC. I’ve never been pregnant, or had a MC. Help! I’ve been looking at progesterone cream, should I try this? I feel all anxious now. It’s taken us so long to get here.