Need your opinion!

My husband and I have been going to weekly family diners for years now (his family, nearby). But lately our baby has been going to bed really early... like 7pm. Considering the dinner and bedtime routine, we'd have to start all that around 6:15 to get him down around 7. Nothing is scheduled that strictly, but that's just happens to be what works right now. Well, family dinner is right at that time. We started just switching off who stays home with baby and who goes, but the family started complaining that we weren't all going. But, I don't want to keep my baby up an extra hour or 2, when I know he's tired and just wants to go to bed. Am I being unreasonable? I feel like they think I'm catering too much to my baby.