SO Family ( venting ) help me 😔

does anyone else not get along with their SO family ? how do you as a couple deal with it ? like at family get togethers and holidays ..
i just had a baby with my fiancé and having my baby made me realize i don't like his family they're cool and stuff but they stress me out and i don't feel comfortable going to their get togethers . our families don't socialize either . i didn't even want to invite his part of the family to my baby shower .
he doesn't get along with my family either like i get we're boring and don't go all out for the holidays or special occasions like his family but it's my i like being with them .
i'm at the point to where when the holidays come around we go our separate ways , he goes with his family i go with mine . 
i've told him multiple times this past months i don't want to hang out with his family he still makes me go with him and he keeps bringing them to my house knowing i don't like them . i'm not gonna like i do have fun with a couple of his cousins that we hang out with but that's it .
i don't even wanna throw parties for my baby because i just get so stressed out and get anxiety just thinking about having to invite his family . i can't do it anymore i don't know what to do . and i feel like if i were to tell him this he would just tell his whole family .. 
 i just need opinions or a way to try to talk this out with him ..