Staying out

Do anyone keep their babies out pass their bedtime on weekends? I am having a dilemma with my husband and the in laws. Just about every Saturday my husband wants to go by his parents with our now 4 month old. ( they live about 30 minutes for us) I usually will go but only want to stay for 5 to 6 hours ( leave by 7pm because our baby goes to sleep 8:30 / 9 pm and so we can do our bedtime routine). Well my husband  want us to stay until 11pm ( we will leave our house around noon --a full day of work basically). So , now we drive separate cars so the baby and I can leave at a good time. Her sister made comments like ( toward my baby) : maybe one day u will hang out with us. Am I being to strict on bedtime ? It just my baby wakes up every hours and is very fussy when the bedtime routine is broken.