Dating my ex-husband and navigating his toxic family

I’m dating my ex-husband. A big factor in our marriage failing the first time was his family. He has 3 siblings and his parents, and all but one of them didn’t like that I was a single mom. His family was awful to me and not much better to my son. We ended up eloping and didn’t even tell them until afterwards. I’ve forgiven them because I know they have issues but I don’t want contact with them.

They’re very concerned with appearances ie: not wanting him to marry a single mom. I went on to get a medical degree and a professional career and a beautiful place to live, all without help from my ex-husband or my son’s father (my family is supportive). Meanwhile him and his 3 siblings are in their mid 20s to early 30s and are ALL unmarried, childless, and living in their parents’ home. The parents are wealthy but I’m now more successful than any of their kids, and they are very superficial. The mother is mentally ill and an alcoholic but completely untreated and in denial, and she goes to great lengths to try to control her adult children.

I haven’t spoken to any of his family in a few years and I don’t want to start contact with them. I love my ex-husband and we’re doing what we need to do for our relationship, but how do I navigate this family situation? He’s very close with all of them and I am very close with my family. I used to put myself through the torture of going to holidays and weddings with him to be treated like shit by them, and I have zero desire to do that. I’ve told him as much and he said we can do whatever I’m OK with including him coming here for holidays with me and my family and then going to see his family by himself the next day etc.

Am I being unforgiving or should I stick to my guns?