Engaged but family drama

I’m engaged (and have been for a year and a half) to my son’s (3 mo.) father but there’s some drama in my family in regards to him and he doesn’t want to spend time with them as a result. He still drops my son and I off and has no issue with our relationship with my family but doesn’t want to include himself in it.

Anyhow, I want to get married but I know it wouldn’t be a special day if everyone was there and drama would be caused but I’m also feeling hesitant to elope and potentially hurt my family’s feelings...

Any advice?