Sibling problems😕

So Ive been dating my boyfriend, Pete for about a year now! And before we ever were talking I thought me and his brother,Jay  were hitting it off but he turned out just wanting a booty call. At this point in my life I had been only thought of as a sex object by any of the guys I hung out with singlely before this so I figured that was all that guys wanted from me and I didnt care. Well anyways Pete and I started to date and he brother started to tell on us and I pretty sure he hates me. Like anytime Im around when he's around we dont look at eachother and he never talks to be or pretends Im not there. Also all the mean things his brother said about me has been carried on to one of there best friends Ryan. So now its always ackward. But I know Pete loves me and I love him. So should I just ignore them being imature and have to deal with that forever in our realtionship or should I eventually end our relationship. I do truely love him I am just wondering if its easier to let him go so that way he doesnt hav to deal with his brothers and ryans crap and also since he just started university so that way he can hav the single experience with his friends n brother.