Am I being too sensitive?

Isabella
Last year, I was in a really bad place where this guy (who was not my boyfriend) put me through some prolonged mental and emotional distress. It was to the point where I was extremely tired and tense and stressed every day for several months. I finally ended the whole crazy thing and am trying to live my senior year of high school as peacefully as I can, considering I still have to see the guy every day (we do the same extracurricular). 
My little brother's a freshman this year, and as to be expected, we butt heads, but I always thought we had a pretty decent relationship, considering I stuck by him through all the crap he went through with his (now ex) girlfriend. 
Bottom line is, I'm really bothered by the fact that my little brother is really friendly and familiar with the guy who was essentially my tormentor. Should I let my brother form relationships with whoever he wants to? Am I being too sensitive about this whole thing? I can't help but be stuck on the idea of familial loyalty, but I need some advice on how to go about this. Thanks.