Is this toxic?

I’m pregnant and me and my boyfriend are both 18. His brothers drink a good amount. He wants to drink when he turns 21. Is it toxic for me to say i don’t want him around the baby if he’s going to drink? and that i don’t want to be around that either? My sister is an alcoholic and i know that drinking doesn’t have to be addictive but i just really don’t want myself or my child around that. Is this toxic? please leave advice.