Ex boyfriend/baby daddy vs boyfriend

Okay this is gonna be a long post but I need some advice.

So I've got a baby from a previous relationship me and the baby daddy let's call him Y were together for 3 and a half years but he was controlling and we both cheated and broken up. We have been apart for 3 years and I've got a current boyfriend whom I've been with for a year.

But here's the problem... my current boyfriend let's call him X is very controlling, wont let me have any Male friends or go out with any of my friends. But I've been talking more and more to my baby daddy recently and I've noticed that I still love him. He's no longer like the way he was, he's been on medication for his anxiety (which was the main cause to him being controlling) he has been seeing a psychiatrist who no longer sees him anymore because he is mentally stable now. I can see he's changed and my love for him is very real and very strong.

However, my baby absolutely adores my boyfriend (she is 5) and I dont want to hurt her by breaking up with him, I honestly have no clue what to do. I feel horrible for talking to my baby daddy while being with my current boyfriend but the feelings are so real and so strong I just don't know what to do. Any advice?