BF very mean to me at random times

Okay this is long, please bear with me: I just started dating this guy (we used to be good friends back in 2007 for about a yr & then lost contact). He found me on social media rec & we started dating in April/May of this year. A little about him: he's only 4 years younger than me. He has an ex-wife (married for 5 yrs) & he has 2 kids with her. He has visitation with them every wknd. Long story short he got his own place (that's pretty far from my parents place, even after I voiced my opinion & that I wanted to live closer to my parents/job at the time. I was commuting 4 hrs every day, 2hrs there, 2 hrs back). He asked me to stay with him. I knew something was wrong (after I lost my job 2 weeks ago). He asked me to fill out my own separate 2 yr lease (which I never did & didn't agree with at all. Made him know that I wasn't ok with it). He gave me my own set of keys (so I can come & go as I please). There's a few major problems with him. he says I'm pretty but that I "would look so amazing with makeup on". I don't like makeup; won't wear it for anyone even if u r Prince Harry. he expects me to do almost everything, as far as chores are concerned. He acts controlling (like for example when I have to go home to spend time with my family/friends & my dog he makes a big deal about it every time. told me: "take care of what I have to take care of during the week, not on the weekend").. then he acts like he doesn't want me there at times, saying I don't have to accompany him everywhere that he goes. I admit I like to be around him all the time. But I've started to keep my distance as of 1 week ago. He works from home & honestly I haven't been spending time with him a lot lately. We got into a bad argument this past Sat night because I wanted to go home & see my fam. A major issue I have with him is that he never communicates back with me (say I call him. sometimes he doesn't even call me back until several min/hrs later ). or he won't even bother to send me a text back ever. So I stopped doing both unless I absolutely have to speak with him. I have trouble hearing sometimes & he called me last night asking me about some job that I had applied to. I said I'm sorry I can't hear you. he called me retarded & then apologized right away for saying it, Only after I called him out & said I wasn't retarded & that he needs to stop behaving like this. He's mean at random times for no reason & says he doesn't trust me. Says he wants to go thru my phone but I don't let anyone look at my phone, not even my best gf. My question is should I break up with him? he's charming when he's not mean but honestly I can't take this anymore