Help!

My boyfriend thinks that I'm doing things with his brother when he's not in the room truth is I don't even like his brother and we only talk when he's not in there if that! I think because he's worrying it will break us apart? Help!!