Age gap relationship

I'm currently in a relationship with someone MUCH younger than me who so happens to be friends with my son. My son is 17. His friend is 18 and I'm 41. Before anyone freaks out nothing romantic started until he was an upper 18. Like he's about to be 19 now. Me and my husband are going through a divorce and he was a shoulder for me to cry on and we became close and he has treated me better than any man. Everyone has found out and are treating me like I'm some predator. My son was angry and said how could I do this because his friend lived with us and used to call me mom because he had such an awful mother. I'm not his mom though and never was. My son moved in with his dad and his dad is gonna try to use this against me in court to get full custody of the kids by implying I'm a groomer because he has lived with us since he was 15. I didn't groom him. Nothing sexual took place while he was under 18. And now my bf is starting to question things and saying he really likes me but doesn't want to ruin his friendship with my son over this. My mom is saying I shouldn't risk everything for an 18 year old. Especially one that's questioning the relationship. I don't think I'm risking everything. He's legal age. I haven't done anything wrong so I don't see how this could be used to claim custody the kids. My son will eventually have to get over it because his friend is a legal adult and I am too. I don't understand why I should have to put EVERYBODIES happiness before my own.