Partner says I’m unfaithful for no reason?

I’ve almost lost all trust with my partner after him disappointing & doing things behind my back, recently it has gotten worst (within the last year or so) but anyway, I’ve never given him a reason to doubt me whatsoever for 6 whole years. Lately, he has been throwing remarks, maybe sarcasm with a hint of seriousness about me being unfaithful, I went out with my girls a few weeks ago and the next day he’s asking me what I’m doing in every photo or who was next to me (most photos were ransoms behind me or walking past etc) then went on to say because I wore his ‘fav’ underwear I must’ve cheated? First he said it jokingly to my gf, then he had an attitude all day & kept mentioning it to the point we argued. Then today my gf told him her partners brother is coming to stay for their wedding (next week) to which my partner replied saying “Yeah this bitches new man” talking about me whilst I was over with my kids, I did not hear this at the time. I’m so mad! If I had put him through what he’s put me through I wouldn’t be okay with being spoke to like that but I’d understand, but when it’s HIM who has done that to me then treat me as if I’m the unfaithful one? Anyone know why he would be doing this? Last thing I want to do is be with someone like this esp after what he has done, it’s really bugging me out heaps.