Partner doesn’t like my family - can it work?

My partner doesn’t get along with my family at all, to the point he won’t come to any gatherings and he will leave the house before they come so he doesn’t see them.

He worked for my dad a while back, things were very tight financially and he was commuting back and forth to help him build a house in the wintertime. He didn’t get paid for the work until a year and a half later - and he’s been very upset since then.

I don’t want to take sides, while yeah- it was shitty what happened (and I was telling him not to do it the whole time) , my dad wasn’t trying to use him and the work was propositioned as “help”.

So I’m kinda stuck, we have a son together and he’s fine with him seeing his grandparents but it’s really hard every time I see my family bc I feel sad that he’s choosing not to be apart of it. Especially since he has no family where we live.