I need advice this is going to be a long post

Last year me and my so had a fight and his mom got involved talking about calling the police on me. I haven’t talked to her since and really didn’t plan on doing so anytime soon. My SO is trying to force me and his mom to talk and every time we go around her he gets mad when i don’t say anything and I rather just stay in the car or off by myself. Every time she hears something dealing with us or me she always tell him that he needs to leave me because I’m just using him. I’m getting really tired of it and I really don’t want to leave him behind his mom but he really not giving me any choice