I need Advice on a messed up situation

So me and my partner were togather for 5 years! We broke up for a few months.. in that time I saw my ex partners sisters ex bf out named frank. They were togather for 4 years.

We confinded in each other a little bit as he got cheated on numerous times and my ex partner was abusive towards me.

We hung out for a good month or so..

He then also was abusive one night when he was drinking, so I gave him the flick and then he went back to my ex partners sister.

Me and my ex partner are now talking again and tried to sort some things out.

Problem is, frank has been telling my ex’s sisters shit and making up lies about me on what I did and said.. then the sister goes and tells my partner.. then they all kinda gang up on me .. last week on Wednesday that happened when they went for breaky.. there now going for breaky again tomorrow and my anxiety is so bad :( frank makes a lot of shit up and then my ex partner and their sister kinda gang up on me and starts fights with me.

I know I shouldn’t of slept with my ex’s sisters partner it was a bad mistake.. if I could take it back I would.

What do I do :(