I really need advice

My boyfriend wanted me to go visit his family they live far away. I said no because I don't have a good relationship with his mom. She is very sneaky and always tries to talk my boyfriend into breaking up with me. I always try to be nice to her when she calls just so he is happy. But he told me if I don't go he won't break up but we'll live like roommates and he doesn't know if he can forgive me because i disappointed him. He is really trying to manipulate me and I don't know what to do. I'm not going to go because if I'm 6-7hours away what am I gonna do if she starts an argument and kicks me out l. Because my boyfriend is not gonna be by my side thats for sure. I'm so sad that he is that way. I understand it's his mom. But if my mom would do these things to him I would defend him and be by his side. I don't understand why can't respect my decision and accept the fact that I'm not dating his mom, I'm dating him. I try to be nice and be good enough for him all the time. If his mon calls, visits I'm nice.