He wants his whole family to stay with us...

We're soon going to move into a 3 bedroom 2 bathroom house. My husbands family is from Virginia, and we've been urging them to visit! However my hubs has revealed that he wants his mom, dad, and 3 brothers to stay with us. His brothers are OUT OF CONTROL. They're 12, 15, and 16. They've thrown shoes at me before, hit me, and slammed my hand in a door. I told him I'd love for his parents to visit alone and they could stay with us, if he wants them to stay, but I don't want the boys in our new house to destroy it. On top of that, I don't want 5 people staying over in our small 3 bedroom house. I need a little privacy at times. Am I wrong for saying this?