Can i get in trouble for allowing my kids father...

To have smoked in the house when we were together? Really i dont need judegement i know i did wrong and i regret putting a man first before my kids or my values. I didnt agree to it first but we would fight and argue about it sooo much that i gave up. He would do it in the backroom and had to have the windows open and a fan. It still made me feel guilty but i let it go. Now that we arent together i am urging him to never be high around them alone (i have never let him smoke and taken care of them alone being high) hes so pist about this and saying im just being a bitch because he left me and how did i allow it when we were together but now its a problem. I told him it was a mistake and the problem was i was always there to watch my kids now im not. Really i know many people smoke it but he abuses it so much to the point where he doesnt have a job and would refuse to go to work because he got lazy and high. Its gotten in the way of his responsibilities and would always spent so much money on it when we were so broke. One time he spent his last 20$ on it and we didnt have formula for our son that night and it was so fucked up. He becomes so agressive or hard to deal with if he didnt smoke. One time he apologized for lashing out on me and it was only because he hadnt smoke and lt got him in a bad mood.

He doesnt plan on stopping or even cutting down. He said it isnt my buisness and im not going to tell him how to raise his kids. Im scared of taking this to court and throwing him under the bus but everyone is saying hes made his bed so let him lay in it.

Im also worried he will tell the courts that i allowed the kids to be home when he was high. Can i get in trouble? Would they take my kids away from me? I live with my parents and i work and dont do any drugs.

Any advice?