Boyfriend wants me to cut off my brother

My brother was diagnosed with narcissistic personality disorder 6 years ago. I don't remember him ever doing anything extremely terrible or abusive. More so just being an asshole. There was a year when it got worse and that's when my mom said either go to therapy or I'm kicking you out. He was in his 20's. He eventually got diagnosed and has been in therapy since. He's doing well. He's managed a relationship. My brother's thing is one of the biggest things in therapy he's taken to heart is if you don't have anything nice to say don't say anything. My brother has to just walk away from situations if he feels like he's about to be say or do something really shitty. It could be nothing that triggers it. We could be talking about food and he will randomly say "I don't really wanna talk anymore." And just leave. Like my whole house. That's what he has to do. My boyfriend has just gotten annoyed with my brother because my brother does that a lot with my boyfriend. The thing is my boyfriend brings up topics that trigger a lot of his bad narcissistic traits. Like making fun and talking shit on someone. My brother has an extremely inflated ego so if he gets started it all goes down hill. Part of him learning to do better is not participating in that because he knows it feeds his ego. So he... Leaves. My boyfriend has been annoyed with that and asked my to just drop my brother. I asked why and my brother said "He can't even have a conversation with me. Why would you want to associate with someone who clearly doesn't like me " I said he's taking that way too personally and he does that to everyone and my boyfriend says he does that with him more and it's so rude to walk away while someone is talking. I said it's better than what he says or does when he doesn't leave situations. My boyfriend started accusing me of also being a narcissist because I won't cut my brother off and why do I want a relationship with a narcissist anyway. I said if my brother was still acting the way he was when my mom almost kicked him out we wouldn't be talking. But he's saying if I don't cut him off I'm choosing my narcissistic brother over him and am a narcissist myself. All this because he doesn't like that my brother walks away from conversations.