should i care?

my boyfriend and i have been together for 8 months. throughout these months there are certain girls that he knows i don’t like him talking to. and vice versa with me. i totally kicked out the boys he doesn’t like and never talk to them anymore, but he won’t do the same!! he says he’s just friends with them and that i shouldn’t worry. am i overreacting? should i just let him keep talking to them?