Selfish of no?

Alright.. So I need some other opinions. My boyfriend has a friend that I ironically was friends with during high school. Around our senior year he started shit talking me for literally no reason. It's now been 2 years since high school and Kyle and I have been together for a year and a half. He continues to be friends with this boy. Even though his "friend" continues to call me "bitch, ain't worth anything...blah blah blah"... As I stated for no reason at all. Do I have a right to be upset with my boyfriend for continuing to hang out with him? Or is that selfish of me? I'm not sure how to feel on this.