Partners fam inviting him to do drugs

I really cannot stand that my partners family always offer him party drugs and weekend mushroom trips on a yearly bases. My partner used to do it with them before our daughter was born I've done it once and really enjoyed it but I just don't like drugs. I just told my partner not to take the offer of doing drugs im a couple of weeks time with his family and he got defensive saying I am lecturing him on what his choices are? We have a 2 year old daughter and I've literally watched his 17 year old niece say she can't wait till she's allowed to try mdma with them and his 10 year old nephew saying he's always wanted to try helium (is that phase 1?) I don't know but from my perspective and from what my partner has told me it has been a chain reaction of adults and children in his family and I see it this way. I don't want my daughter growing up thinking that's normal and I don't want to be a drug using family. But he's making me feel like I'm controlling and I'm in the wrong here. Am I in the wrong? Should I not be interfering in what he wants to do? I just feel like we come from different worlds where it's to do with drugs he's grown up seeing it as normal and I've grown up not seeing it at all. And I don't like being around drug users when they are on drugs or coming down from drugs it's just not me or what I stand for.