Should I drop male friend?

My boyfriend of 7 months is uncomfortable with a male friend that I've had for a while. I've known the guy for over 10 years. We don't hang out. Usually he calls me to vent about his life or ask for advice. I talk to him in the phone a couple times a week. I see him on very very rare occasions.

In the past the guy had expressed some interest in trying to pursue something with me but the boundaries of our relationship has been made very clear. We both understand that we are only friends.

My bf has trust issues that come up sometimes and sometimes thinks that I'm not honest about my relationship with this friend. It didn't help that he heard us talking on speaker phone a couple weeks ago (he wasn't eavesdropping I knew he was right next to me) and the guy made some joke about how he should be able to take me out on a date. I admit the guy was wrong for the joke because he knows I'm in a relationship but at the same time we both know the joke was a joke and nothing was expected.

My bf felt disrespected and thinks that in order for us to continue building our relationship that I should drop this guy friend. He says not just because he's a guy but because he's a guy who has shown interest in me. My thing is that seems rude to just cut someone off. I don't know if I could do that. I have a conscience you know?

I hate that my bf felt disrespected and do not want to jeopardize our relationship because I believe it could be really good. Any thoughts /comments will be helpful? I don't know.