What shall I do

Okay I been with this man off and on for 4 years as just friends so recently about 2 mons ago we decided to make it a relationship but we came into the relationship talking to other people’s I had my ex of 18 years and he had a couple of fwb so now there’s trust issues between us I stop talking to my ex and he said his did the same but I keep finding hidden texts notifications on his phone but once I go to messaging there’s nothing there we both have each other codes to out phones he doesn’t want to have sex nor cuddle for no more then a few mins we don’t talk unless I’m piss off now he say I’m smoothening him cuz I want to be with him every time I ask him about anything he gets mad and try to flip it on me as if I’m the bad guy I really love this man and honestly I don’t know what to do