Can your kids get taken away?

Was wondering if you smoker pot & got caught if your kids could be taken away? I wouldn’t be smoking around them, & it’s not yet legal in my state but you wouldn’t get arrested for it either they’d either let u go or take it away.