Just had my baby

First time mom & today is my boyfriends grandmas birthday, i just had my baby 12 days ago and they invited us to go out to eat, but the dr told us not to take the baby out at all, to always stay home till hes a little bit older, would it hurt him if we take him out? Or do we just stay home? Help pls