Should I avoid caffeine during the first trimester?

I know you can have 200mg a day, but I also read that you should avoid it all together during the first trimester. I’m only 4 weeks so still very early and risky and I don’t want to do anything to potentially hurt the baby, but goodness I’ve only been off caffeine for two days and I have a nightmare of a headache. I will soldier through if that’s what is best, but its worth asking!