I don't know what to do.

The problem is I've been in a realtionship with this guy ( lets just call him A so i can type this story easier)for the last 3 months or so. Just last week he took me to meet his family.His brother (lets call him B) is two grades above us and I had a small crush on him in 6th grade. He also found out about 2 weeks before school ended which was fine cause I wouldn't see him for another two years and he would of forgot me by then. Skip to 4 years later when his brother and I are sophomores, he still rememembers me when I go to meet my boyfriend's family. Now, "A" knew I had a crush on his brother before, but was totally fine with it as I told him when we were still friends. Now everything is going great and I begin to really like his parents, but B has been really quite and just looking at me really weirdly. I just brush it off. Later I go to get a drink of water but B follows me. He begins talking about how it's pathetic that I'm messing with A just so I can get "closer" to himself. I tried to explain that he was a small crush that I got over about 5 days after school, and how I actually really love him. He wouldn't listen to me and just said I needed to break it off with him or else he'll tell him my "plan". A and B aren't that close at all but I still feel like there is a small chance he would side with his brother and think I used him to get closer to B, while in reality I really love him and don't want this realtionship to end cause of his stupid brother.