Boyfriend troubles

My boyfriend recently went out of state to visit some friends for the weekend. No problem with that of course! But then he tells me they're into drugs, and I immediately feel a bit uncomfortable naturally. My family has had troubles, not to mention my boyfriend had an addiction when he was around 13-14 (didn't know him then). I express that I don't particularly like him being around it and he gets pissed saying it doesn't matter cause I'm not there, it's his friends, why should I worry about it? I'm upset. Next day when he gets home I confront him and explain that it's normal for me to care. He says he likes being around the drugs, not doing them, because it's fun. Would anyone else be worried? I trust him not to do anything BUT simply being around illegal shit isn't always the best choice. Maybe I'm overreacting because of personal experience... I dropped it with him because he has his feelings about it but I want to know if I'm in the wrong or what other people would think.