Dinner with child’s father family

I live in a different state than my child’s father and his family. His mom and sisters came to visit my daughter and wanted to go to dinner. I told my fiancé this and he’s not feeling it at all. He feels like I should have a very surface level with them because that’s how he is with his child’s mother and her family. I don’t really interact with my daughters father at all just his mom, grandma, and sisters since they actually take the time to spend with my baby. I don’t feel wrong for going to dinner, but maybe y’all can give me your thoughts.