Is it rude if I tell my neighbors I'm sleeping when they want to come over?

Maikel
my neighbors have 4 kids, and from the moment they see my boyfriend's truck pullout the driveway in the morning they want to come over. I love having them here, but I would also like sometime to myself. is it messed up if I don't open the door when they knock and later say I was sleeping?