Bringing Newborns Around

We are only TTC and my husband, who has a close relationship with his coworkers, is already talking about me bringing our future baby in during maternity leave to meet his coworkers. Although they are super nice, they are always sick and generally, an unhealthy group of people. Im not even pregnant yet and I am not comfortable bringing a newborn, who hasn't had any vaccines yet, to this petri dish of a place.

Am I being dramatic? How do I help my husband understand why I'm uncomfortable with the idea?