Such a long story...sorry

I have 2 kids with my ex. We split up because he was using drugs and no matter how hard I tried I couldn't help him and I didn't want my children raised around that. I would work 12 hours a day and my dad who was injured babysat while baby daddy slept at my dad's house. Well I finally left him and fell in love. I have an amazing man by my side. Fast forward to now. My ex hasn't seen the kids for months. He says he quit drugs but doesn't pay his child support. He never ask to see the kids but I offer for him to come see them since he is their dad. My daughter was 6 when I left him so she remembers him. He always has an excuse not to. This makes me and my current guy fight. He hates that my ex treats the kids this way. Like over Thanksgiving. I told him he could have them at his parents. he said no and his parents only wanted my daughter not my son so I said no. It's not really fair to my son. My current guy hates seeing the kids depressed and my kids do call him dad. I think my 2 year old knows only him as dad really. I'm just not sure if I should listen to him and stop all contact, I don't want to fight with him but I don't want to be the mom that forced the bio dad out of the picture. I want me kids to see that I tried and what happened is all on him. I just don't know what to do sorry it's so long. It's also on Anon because I know I'll probably get judged for being with a drug addict while I had my kids. I didn't leave right away because most people need support. After so long I couldn't do it anymore though. I was with him for 9 years. He told me he wasn't on anything, it seemed weird he was always broke. Once he started taking my money I knew and I cought him. He would take my daughter will him to make deals. This is why I said he gets them at his mom's. My daughter has said how sad it makes her but she doesn't cry about it anymore. I might eventually tell them the truth but their step dad or just dad as they know him is more of a dad than their real dad. My dad is a father figure and helps more with them even though he's back to work now. Anyways. Any advice?