No more coffee???? Insight please

Kristen
So yesterday at my dr appt my boyfriend asked the dr if it's ok for me to be drinking the HALF caffeinated coffee I've been drinking. I was taken aback when he said no because caffeine was related to birth defects. Now I have read everywhere that coffee or caffeine in moderate amounts poses NO harm to the baby, and I switched to half caff because I was a huge coffee drinker prior. Has anyone been told this as well? I'll drink decaf if I have to, because I want what's best for my baby, I just don't get why all these apps and sites state 100-300mg a day (1-3 cups) is fine if really it can harm the baby. Any feedback would be great 🙏🏽😁 Thank you ladies!