Wondering what to think or do about this....

So I've been together with my kids dad for 7 years, but ever since we've been togrher he's been sneaking back to his ex wife for sex along with other girls & everytime its brought up he chooses to get mad & flip the situation on me & not him. We talk but lately our convos are short no more than a couple words & its pushing me away .....