1st Birthday Party Advice

My son turns one on 5/22. Since it's falls on a Monday, we will celebrate the following Saturday which is Memorial Day weekend. I decided that I would just cut a cake & Bbq on my balcony, since most ppl go away.  Problems is everyone says they are not going anywhere and want to come over. My home is a big 1 bedroom and even though I'm going to put things in storage I'm so scared it will be over crowded. Is it wrong that I hope ppl will cancel?