Spouse annoyed we don’t eat together

My husband has been amazing and so supportive while we are home with our newborn and toddler. He does basically all the cooking and toddler care. But he is getting annoyed when I’m holding baby for a nap at an inconvenient time (like at breakfast) and the toddler would wake baby up so I choose to eat later. Is there anything I can do?? He’s a really light sleeper and doesn’t want to be put down but my husband is sad we don’t spend the time together and it’s not like I enjoy this either- but baby will get overtired abc scream if we just throw his wake windows off randomly.