He didn’t let me to check his phone until we move

I have been with this guy for 4 months I’m trying to be honest with him and keep with him we just have sex 1 or 2 times at the week he said because he is working out at the gym blabla the things is I ask him if I can check his phone because he has his phone in silent all the time he said NO he told me that he has a lot girls like a friends and they are texting him all the time but he said that I’m his girlfriend and I’m the person that he is taking serious I already met his family and he text me and call me every time that he is not busy he pass with me his day off so we are most all the time together so should I have to keep going with this relationship or should I just leave because he is still talking with another girls?