Venting about my bf

So my bf and I have been together for about 2 years (we’re both in hs) and up until this summer, we would exclusively hang out at his place. I was over in June and he and his mom got into a huge fight (I won’t go into specifics but it was in a sense about me), and that was the second time they fought while I was over. I am awful with confrontation, and seeing them fight was very overwhelming for me considering I got dragged into it. Since then I haven’t gone over and he always comes to my place. His mom apologized to me and we’re on good terms, but I still feel uncomfortable around her. Today my bf was supposed to come over but last minute asked if I could go to his place instead because he was really tired. I told him that we could just see eachother another day and he should just get some rest. Now he’s upset that I won’t come over, saying that he always has to commute to me and that it’s not fair. I reminded him that I don’t feel comfortable being at his house, and his mom has expressed she’s not really comfortable with it either. He’s telling me that I’m being unreasonable and that I can’t avoid his house forever/that I just shouldn’t care about what his mom thinks, and I feel like he’s just being inconsiderate and kinda selfish. I’m mostly writing this to vent, but if anyone has advice or anything please share