Father’s Day dinner

So my parents want my daughter and I to come over on Father’s Day for dinner, but I’d rather be with my boyfriend for Father’s Day doing our own thing (he agrees) and he said he doesn’t want to be without our daughter and I that day. My parents don’t like my boyfriend and they don’t want him over. I told him it’s his day to choose how he wants to spend it, especially since he gave me such a wonderful Mother’s Day, but I just know my parents are going to be upset that I’m not seeing my father on Father’s Day. Am I wrong for prioritizing my boyfriends day over my fathers? Or should I still go to my parents dinner that day?