Need help , thinking about taking away fathers rights?!

Okay to make the long story short I have a beautiful 4 year old daughter from my ex.

He was there emotionally ( not financially ) for my daughter until she was 2. He got into crimes, ( stealing and robbing people) until one night he and his friend went out and ended up killing someone. He didn’t but his friend did. He ended up getting put in jail 2 years ago and is awaiting his sentence which will be 20-40 years.

At the time I didn’t realize the severity of the situation and how unhealthy it was to be with him, he kept making me seemed crazy for wanting him to stop.

Anyway fast forward to today, I found a wonderful man who treats me and my daughter with the upmost respect and me and him are now expecting a girl :) .

He treats my daughter like his own, he gives her everything , and always refer to her as his daughter.

I don’t know what to do in this situation but I feel like she looks at my fiancé as her father and I just feel like she should not even go and visit her bio father in prison.( he’s behind glass ) She was so heartbroken and I don’t want my daughter to be in an environment like that. She doesn’t talk to him much on the phone and I don’t want her father calling me.

Would it be wrong if I could remove his rights? I don’t want to be the bad guy in this situation but I just don’t know what to do. And by the time he’s out my daughter will be 20+ years old.

Please help