Minnesota laws

I'm 25 weeks today and I'm afraid . I lost 30-35 lbs in the beginning of my pregnancy and I still smoke with my boyfriend a couple times . I'm afraid they'll take my baby. I'm afraid to be judged but I've done my research. I always make sure stuff with him is taken care of before myself . I make sure he's okay & everything I know I shouldn't feel bad because it helps with all the symptoms but everyone around me makes me feel like I should. Any answers on what to do? Or if my child will be taken? Anything helpful please !!!