Relationship question

I constantly feel like I’m stuck in the same situation as always. Before I came along my baby daddy had some not so good friends. They are into drinking and hardcore drugs. Bad influences. He finally broke free from them. I’m not trying to be controlling at all but I just don’t need him getting drunk and drugged up. Our relationship has improved so much since those people are gone from his life. Anyways, he blocked them and what not. However now, these people are contacting him from other peoples phones, giving out his number and just trying their best to get his attention. I know he really wants this family with me but I refuse to be apart of drugs and alcohol in this family situation with a newborn. I also feel like when we’re not together he might try hitting them up to see what was wrong. He told me they haven’t talked in months so having an emergency I’m not sure if that’s some code words they came up with or not. Like I said I’m not being controlling but I am in control of how my life goes. What would you do in this situation? I kinda feel like running away tbh. I don’t want to hold anyone back from what they really want. He did tell me he tried calling him 3 times and texted from someone else’s number. Which I appreciate the honesty but also don’t want him to feel like having 2 separate lives because he knows I don’t approve of the friendship. If they could have a healthy friendship that’s fine but it’s not. Should I just leave and go about my business or what? I’m really stumped here.