Coparenting with a narcissist advice

So my ex is a huge narcissist and possibly has borderline personality disorder. While we were together he was in the navy & I moved to Oklahoma with him when we got married. He separated me from everyone - made up reasons why he didn't like any of my new friends, accused me of cheating if a friend was a male, made me lie to my parents about why we couldn't see them - and constantly downgraded me calling me stupid & a terrible wife (& later a terrible mother). When I got pregnant he "convinced" me to quit school & then quit my job & even convinced me to give up my cat & dog (thankfully my parents took them).

Anyways, I finally went to therapy & realized he was an emotionally abusive asshole & left. During this time he decided to cut off his own parents & sister as well because they came to my son's birthday party. There's a lot more to this story, but to get to the point:

My son will be doing tball this year & his first game is on a Saturday. I want to invite my ex-inlaws & my soon to be inlaws to watch the game since its the only one they can make it to (they all live about 1 1/2 hrs away). My parents will come regardless. The problem is - I don't want any of them to feel awkward or any drama to happen if my ex goes to the game, too. My fiance has already said he won't go to the game if my ex is there because I've told him everything that happened in the relationship & he does not have the ability to be civil around him (this is childish in my opinion, but I understand why he feels this way). I thought about telling my ex the first game was a different day, but 1 of the practice days is when he has our son so I'm sure it'll get brought up. I feel awful for even thinking of doing that, but he's also not really ever been the best dad. He chose to only take our son every other weekend & my son was so used to only being around me that he used to scream when it was time to go see his dad. I'm just stuck on what I should do. I know his family would be really upset to not get to see him play, but it'll also upset them to see their son there (he threatened to file harassment charges on his mom because she was emailing him trying to see how he was doing). So for anyone who took the time to read my rambling, what would you do in this position?