I need advice. Am i wrong for want my boyfriend to let his friend go for good?

I've been with my boyfriend for a year & a month. I don't like this one friend he has because i knew his friend before he did. We had slept together but somewhere after there he was really rude to me for a long time & has called me the N word & was being racist towards me. & even when I'm around now & my boyfriend is there it's like he always has something to say. The one day he said something & my boyfriend said he didn't hear it. I don't want to ask him to not be friends with him but they aren't really close friends they just used to get high together all time & even now I'm not comfortable with him around when I'm there or not. My boyfriend doesn't tell me when he's with him & then i find out & get upset. & my boyfriend always expects me to tell him what I'm doing when I'm with my friends but whenever i ask him what he's doing when he's with him he doesn't tell me. I really don't trust this guy because of the past & i can't let things go so i need help. What would you do? Thanks