Should I leave or work it out.....

So I've been married for 2 years. Very rocky start. My family and his family don't speak due to many disagreements. 
​We promised to keep to ourselves and each of us still have a strong relationship with our own families. So we can't go to family for help... 
​We have been having many issues and for the past 6 months our sex life has diminished. Once a month if that... 
​ he is saying your not affectionate and don't show me love and your always upset that's why I have pulled back. 
​I say we don't have a trusting relationship anymore, we are more comfortable than happy but we love each other and when it comes to the crunch, none of us can walk out. 
​Recently I have seen a conversation between him and another girl. He was affectionate with her and spoke to her like he was very interested told her how much he cares for her and needed her... 
​I confronted him and he said I know inm wrong and I'm sorry and it was only because you don't give me attention and I long for that feeling etc etc
​What should I do ? 
​We have agreed to give ourselves a couple of hours to cool down and think of what we really want to do. 
​I need advice, what should I say to him or do 
​Am I to blame for not giving him love, attention, affection and so that's y he's resorted to this ?