I just need major advice! Right person wrong timing.

Okay so 4 years ago my friend talked to this guy for a couple weeks and whenever she would see him I saw him mostly at parties. But him and I since then have became super close friends we even both had kids around the same time and we both aren't with the other parent so we became even closer one night we were just hanging out and I don't know how it happened but it felt so right when it happened and we've been having casual sex ever since he's told me how long he's wanted to kiss me and how much he likes me blah blah blah but we've been keeping that part a secret. Now why is it a secret? 3 years ago I had a fling with his brother! Omg 🙈 I can't even. Plus my friend is still obsessed with him apparently how I found that out is because I was telling her how him and I have been talking more often now to at least get that part out of the way. And I told her how him and I went to go get tacos last Tuesday. She flipped out on me and started crying. I asked her why she was crying she still has feelings for him even though they barely talked in the first place it was a small thing nothing to him everything to her! Him and I because of these two people decided to keep it a secret. He's the right person but this is wrong timing I feel like and I have no idea what to do. The sex is mind blowing and he's amazing! But the other people involved from our pasts is a lot. Please just give some advice! I know how messed up the brother thing and friend thing is trust me.