Boyfriend problems

Okay so my boyfriend is highly insecure and I respect that because I understand that he's been hurt. Lately he's been joking that his roommate is my "boyfriend" he keeps threatening to end the relationship whenever I talk to his roommate or hang out with him even if my boyfriend is in the room. I just feel so hurt that he can't trust me the way I trust him and I need some advice on how to go about this; either with being more patient or how to talk to him about it. Thank you!