Would you lose friendship over relationship?

So I met this wonderful agent when he was out doing something. I didn't think I would ever see that guy again. Well when I was in a business meeting, homeland had to attend. I didn't expect to see him again because there are so many agents around here and he sort of seemed busy when I said hey. Anyways my friends dont wanna be around me because of his profession. Am I wrong to drop them?