Custody issue..?

Messy situation..

I got pregnant 2 or so years ago by a guy I was seeing who moved across the country. We broke up and he wasn’t involved.

I started dating someone when I was pregnant and ended up being in a relationship with this man through pregnancy, birth and until my son was 3 months old when we had a messy break up. I was suffering from PPD at the time and my family and others practically bullied me into giving into giving him unofficial 50/50 custody as they didn’t think I was equipped to look after my child.

I’ve since been stable and thriving for a year, in my own place, working and supporting my daughter. I’m still sharing custody and it sucks. Even though he treated me badly my family loves him and backs him on everything, even overruling my opinions as a parent in favour of his.

The biological father has sinced moved back into town We’ve been seeing each other for 4 months without involving my son and now he’s desperate to step up and has been regretful for not doing so in the first place.

What do I do in regards to the custody I have shared with my ex. I don’t want to be horrible and take away my son as I know he loves him but I really don’t see this working out long term and it’s not an arrangement I wanted in the first place. Legally has no right to him but I just feel stuck between a rock and a hard place, knowing what I want to do and what I feel is best for my son will hurt people not to mention turn my family against me. Advice.