Am I being insensitive?

Rebecca

My fiance and I are going to visit my grandparents out of town for Thanksgiving he has never met them due to them living so far away and I have not been able to afford to make a trip down for the last three years. They are getting older, and have been having some health problems as of late so it's very, very important to me that my fiance especially gets to meet and spend time with them. So here is the "issue" his nephews first birthday party is the day that we are supposed to come home but it is in the early afternoon. In order to make it home in time for the party we will have to leave from my grandparents at 6:30 in the morning. This means, only spending one full day with them and missing out on attending Church together (which is a family tradition). My honey tried to explain to his mother that we may not be able to make it and that it's important for me to see them and for him to meet them and before he could even finish she told him she didn't care and we better be there. His sister and sister in law have both told him multiple times "they'll never forgive him if he doesn't come." I understand it's important to them but we live five minutes from the nephew, and he will have many more birthdays. My time with my grandparents is definitely limited. I guess I just wonder what everyone's opinion is?