Would you consider this controlling?

My bf is a good and professional guy. But every once in a blue moon when his closest friends are around (nice people btw) they like to do go out drink and do either cocaine or X. I have a strong problem with such hard core drugs! I tell him I don't want him doing it because I cannot trust anyone under those influences. He says how would I know if I've never done them myself. I'm starting to resent him for falling into peer pressure and I feel he's starting to resent me for trying to 'control him'. I'm at a lost. Am I in the wrong for this? Would you consider this a deal breaker? Thank you ladies..