Coffee and TTC

Happy New Year! My fertility doctor told me to stay away from caffeine while TTC. I have two pregnant coworkers that have a cup of coffee a day. I’m so confused!