Could you be with someone if you hated their friends?

I feel so lost. I’ve been with my boyfriend for 3 years. I love him more than anything. His friends, however, are a different story and have been a point of contention for all 3 years. Really our ONLY point of contention as our relationship is quite happy and healthy aside from that. He spends most of his time with me but I do know he def needs time alone with friends so I don’t mind at all when they hang out. I even cook for them when they come to the house and am very cordial despite me not liking any of them (except one). He has been friends with the same group of people for years and years- like since middle school and we’re late 20s- so I understand the history they have. My issue with his 3 main friends is this: they are lazy, alcoholics, drug addicts, racist,unmotivated, rude, and come over unannounced from time to time which really grinds my gears. They come over even after my bf has ignored their calls. 🤬My bf is amazing but he is slightly passive and has a hard time telling them to be more respectful. Or even just telling them to leave when they show up and we have plans or are in the middle of something. He gets annoyed himself when they show up but still lets them hang out! Sometimes for an hour plus! Blows my mind. Let me end this by saying I know you are probably wondering why my bf would be friends with people like this because rest assured, he is NOTHING like them. He is a huge family person, very respectful, conscientious, kind, helpful, generous, ambitious, motivated and hard working. They do have similar interests such as cars, trucks, motorcycles, fishing, etc but beyond that I can only think the reason is the length of their friendship. My boyfriend grew up and matured but NONE of them ever did. I feel like they drag him down in a way even though he really does limit time spent with them. So maybe I am being unreasonable ? I just feel like all our fights surround them and their behavior. Why would my bf want to be around people like that when he tells me himself how miserable they all are and he needs new friends?! Im at a loss. What would you do? Or do I do nothing? Maybe it’s a battle not worth fighting since my boyfriend is amazing and great to me?