Is it wrong to date my baby daddy's friend?

Here's the story:

My baby daddy and I dated in high school. We were together for 2 years. Broke up when I was 6 months pregnant and living together. He cheated on me. We called it quits and moved on with our lives co-parenting and get along just well. Our daughter is about to turn 7 years old in August.

Now.... His friend and I have known each other since middle school. We never really talked then but knew one another existed. He's a finance manager at a dealership whom I bought a car from a few months ago. I guess with me going up there constantly and interacting with him, he started to get to know me and thought I was "gorgeous" as he likes to say lol.

Anyways, I invited him out for a drink one night. Thought I could treat him since he gave me a really good deal on the car I bought. I really took it as a friendly encounter since he's easy to talk to and didn't think much of it. But he on the other hand, started to fall for me that night. Knowing that he was my baby daddy's friend, I figured he was off limits. That was well buried in my head. Then we hung out a 2nd time and this time he told me how he felt about me. Pretty much told me that he wanted to be with me. I was in shock. I told him it was wrong because of my baby daddy. But then I started to think twice about that.

Fast forward to a few weeks later (now) we decided we are going to secretly date and spend time together to see if we actually develop feelings and want to be together. We don't know for sure how my baby daddy/his friend will take it. They aren't best best friends but have hung out with the same crowd recently. He says my baby daddy is an asshole and doesn't consider him a close friend. I really need some advice at this point because I'm really starting to like him. He's a great guy and he's the only person I've met so far that actually makes time for me. His actions have shown me that he wants to be with me without a doubt. But is it worth him losing a friend? My best friend told me that we should do what makes us happy. My brother also said the same thing. And if so, how do we approach my baby daddy or let it be known?

By the way, I am not using him to make my baby daddy jealous. I don't play those stupid games. There are no feelings left with him and I. We just co-parent, that's all.