visitation

at what age did you start letting your baby stay at his dads house without your for a visit my baby's 5 months and his dad has only seen him 2 times since he was born and in those visits I was there, but he wants to me start leaving him at his house for a couple hours but idk I'm not sure about that is that like a fair feeling?