No more coffee???? Insight please

So yesterday at my dr appt my boyfriend asked the dr if it's ok for me to be drinking the HALF caffeinated coffee I've been drinking. I was taken aback when he said no because caffeine was related to birth defects. Now I have read everywhere that coffee or caffeine in moderate amounts poses NO harm to the baby, and I switched to half caff because I was a huge coffee drinker prior. Has anyone been told this as well? I'll drink decaf if I have to, because I want what's best for my baby, I just don't get why all these apps and sites state 100-300mg a day (1-3 cups) is fine if really it can harm the baby. Any feedback would be great 🙏🏽😁 Thank you ladies!