Your thoughts on caffeine..

So I'm having a hard time with my boyfriends family when it comes to me consuming a little coffee. I buy myself a 16oz coffee once a month but only take about 5-6 sips of it. (Wasteful I know) and now they are all saying I'm a bad mom for drinking any at all and how they hope my Dr is a good Dr. (Was told one cup a day was fine) and how it's going to cause the baby to have problems and all of this. I'm seriously about to lose my cool.