Which family are you closer to in your relationship?

I feel closer to my SO's because my family is crazy, filled with drama & most of them are drug addicts. I've been through so much in my life that i can't wait to move out. Am i wrong for not wanting to spend labor day with just me, my bf, my mom? She wants us to sleep here & have dinner with her. But shes called my boyfriend every name in the book & shes bipolar. Even though she's apoligized i just feel uneasy whenever i have them around each other. My family doesn't bother with each other & no one really bothers with her. Should i give it a go?