New born staying the night out

Me and my boyfriend do not live together we both still live with our parents when the baby gets here he wants to take he/she to spend the night over his house but I’m not comfortable with my new born away from me at night until she’s a bit older. Am I wrong for feeling like this ?