Horrible or worth it!? Conflicted views.

Okay so this is a bit of a long story. 2 months ago i left my now ex. We were together over 4 years. It was mutual we just didn’t want the same things and it just was miserable. He wasn’t romantic only touched me for sex and we fought a lot. I’m a lot happier since moving away and getting a new job. My problem is my best friend is his first cousin and about 6 months ago, I noticed I had slight feelings for my bestie. Of course I never told him, never acted on these feelings because I’d never cheat or cross that line. He lived abroad so it’s not like we even seen each other for nearly a year. He’s back now and we hang out a lot. We always have we’ve been besties as long as I dated my ex and he’s someone I love and trust 110%. We sort of got drunk and both admitted there were feelings between us and tried to make sense of it all. One thing led to another and we Hooked up. I’m not proud of my choices but I don’t regret them. It was incredible, we just clicked and we get each other and we both really really want to see if a relationship could work out. But we both know my ex is angry and bitter and he’s already threatened his cousin that if he touches me he will hurt him which we don’t want. I guess what I’m wondering is, if maybe a few months or a year from now would we be shitty people for being together if I was already dating someone in that family? :( I feel like I’m denying my heart something it truly believes in just for the sake of my exes feelings.