Am i wrong? *Drug related topic*

Matched this guy on tinder and chatted to him for a few days. He asked me what my stance was on weed, I said I don't have a problem with it I only object to it when people drive while high. I asked him if he does any of the harder stuff and he said sometimes. I personally don't wanna be with someone who does hard drugs so I said that's my preference, I'm cool if you wanna do weed whatever but as much as he seems like a decent guy, I don't wanna be with someone who does harder stuff. To which he replied saying that's "cool I wanna be with someone who understands". Should I feel guilty? My ex did drugs and got clean from them but he told me how absolutely hard and life altering it was, it was horrible being on them and he'd never go back. I don't wanna be with someone who does drugs, I've seen what it does to people and drains their money, wrecks their life. Am I wrong for saying I'm not looking to date someone who does drugs?