My doctor told me no caffeine 😩

I caved and drank half a cup of coffee after 13 weeks of none because I've been craving it πŸ˜“. Should I tell my doctor? I'm so scared she'll be upset with me. But I've heard small amounts of caffeine doesn't harm baby. πŸ˜”