I need help

So I told my boyfriend that I would go visit his family with him (they live 6hours away) just so we wouldn't get in an argument. I don't know how to tell him that I don't want to go. I have problems with his mom 'cause she talked bad about my mom, called me and her a wh*re and she tried to break up my family. I really don't want to spend several days at her house because you never know when she flipps out and I just wouldn't feel comfortable. My boyfriend doesn't really understand my problem and says that i should just get over it. But I can't. I told him I would go with him because he would be reallly mad at me if I don't go. But I don't know how to tell him that I don't want to go. Any tipps?