Should I cancel our baby shower?

Okay so my husband's family was complaining that we needed to have a baby shower for this baby even though its our 5th. They say all babies need to be celebrated. So we finally said okay and picked a date. Well we let everyone know when and all that jazz. We'll now his whole family except like one person is now saying they can't make it. I feel like I should just cancel it. Its literally going to be my mom, my brother and his wife, my kids, my husband's cousin, maybe a couple we hang out with and that's about it. I originally wanted to do something after the baby was here but everyone kept pushing. So what should I do?