Vacation without baby?

My husband and I are going away this weekend for four nights, five days. My son will be with my parents who will be staying at our place so that he is in his environment. I'm really excited for a getaway with my husband but I'm super apprehensive and nervous about leaving him now. I'm worried he'll miss us? Or his sleep/nap schedule getting ruined? I dunno. Has anyone left their little one for a few days? How was it? Was their schedule the same when you returned?