Overnight rights?

What are my rights as a mum stopping my child's dad from having our baby over night? He see's our child often and brings him back home to me before bedtime. If he was to take me to court for overnight access where would I stand? I have my own reasons for not feeling comfortable with it, I'd never stop him having our son but day times only.