Getting back with an ex while pregnant..

When I found out I was pregnant me and my bf weren't on good terms anyway, with his drug use, no job and always complaining about life, things ended pretty quickly after that. I've been thinking about getting back with my ex of two years from before, he knows I'm pregnant and is okay with it, just the father of my child is starting to say things like he better not come around his kid, watch what will happen, bla bla. What's your opinion