He wants his whole family to stay with us!

We're soon going to move into a 3 bedroom 2 bathroom house. My husbands family is from Virginia, and we've been urging them to visit! However my hubs has revealed that he wants his mom, dad, and 3 brothers to stay with us. His brothers are OUT OF CONTROL. They're 12, 15, and 16. They've thrown shoes at me, slammed my hand in doors, and just physically attacked me. I told him I'd love for his parents to visit alone and they could stay with us, if he wants them to stay, but I don't want the boys in our new house to destroy it. On top of that, I don't want 5 people staying over in our small 3 bedroom house. I need a little privacy at times. Am I wrong for saying this?