My ex and my new guy live together

So my ex and my new guy live together along with a couple of my very good friends. I go over to see him and my friends after dinner and out of no were he leaves with my ex and texts me saying I was talking behind his back. when all I was talking about was what was happening all week between me and him with my girlfriend in the house. I understand why he would have thought I was being like that. His baby momma does the same thing. but I have a feeling my ex is not helping at all. Am I in the wring and should say sorry even though I already did? Is my ex trying to sabotage any chance I have? Or am I being paranoid?