Advice?

I’ve had these friends for a while, we all fell off for a while because I was “spending too much time” with my boyfriend. Well we all made up recently and we’ve hung out twice. I’m starting to get really uncomfortable around them, especially when one of them told me she did a certain drug once and would actually do it again (not marijuana, it’s heavier) and it’s freaked me out pretty badly. I drink, i used to smoke but cut it out when i met my boyfriend and now i only drink on occasion.. I don’t want to be friends with them anymore and i feel so bad. What should I do? I don’t want them to be mad at me but i just don’t want them in my life anymore.