Need some advice

AssalamOalaikum sisters, I hope to get some advice on what I should do in my situation. my husband is someone who had a terrible past and now is changed for the better. he used to drink and smoke weed but he stopped drinking before meeting me and he recently quit weed after I got pregnant. we live with my in laws and his brothers. his older brother is not a good guy. very bad role model, does hard drugs, drinks, smokes weed and many other unislamic acts. he is in the middle of his divorce because he was abusive toward his wife and cheated on her. today my husband is gone out to watch sports with his friends and brother and I saw on his phone where his brother was bragging about drinking and telling my husband to buy, my husband did not say anything that would make me think he will or will not drink he just joked back with him, didn't say no or anything. when I confronted his he said we're just joking. but I don't trust his brother, I don't know what environment he is taking my husband where he will be drinking. I can't handle if my husband goes back into the life he was in, I married him because he had changed and is a very good husband. he is very caring and a good husband but I don't trust what's going on with him and his friends/brother. he tells me to have faith in him and he is not like that, he said my friends drink and stuff but I don't. I'm very scared I don't know what to do, I don't want my child to be affected by this. I don't want to leave my husband either, please can someone give me advice on what to do. I don't want his sins to affect my child.