Inappropriate friend

So my boyfriend was having a guys' night at his friend's house. He called me to say good night. While we were talking, this friend of his (that I absolutely hate) spanks his leg and says "ask her if she knows this sound". And my boyfriend just casually asked me and laughed. I managed to answer "yeah tell him that's what I do to you". I don't know if I should've said it or not but I didn't want to keep quiet. I was speechless because it's fucking weird. I have met this guy like twice. I barely know him. Now I'm a little bit weirded out because my boyfriend didn't say anything. This guy is like his best friend but still what he did was inappropriate and it made me uncomfortable. Am I overthinking? Was it just a *harmless joke*? And should my boyfriend have said anything?