sneaking him in

So my boyfriend keeps asking me if he can come over in the middle of the night when my parents are asleep. I keep telling him no because I don't want to get in trouble if we get caught. How can I tell him this without sounding like a baby??