Need help/ very frustrated

Okay I need some advice. My boyfriend of almost 4 years does something that really drives me crazy. He will say “something is really bothering me but I don’t want to argue so I won’t say what it is”. He literally will refuse to say what the problem is and he does this often. He did it maybe less than a month ago and did it again today. I have a toddler, I’m a single mother. He asked to see me tonight to spend the night with him. I said, you know I don’t like leaving her off with my mom over night. Then he said “this is getting too much now”. I said well, you know I’m a single mother. He then said “its more than that but I don’t want to argue so I won’t say anything “. I’m like, why do you keep telling me that you have some huge issue that bugging the hell out of you, but seriously REFUSE to tell me what’s wrong. But then will say come over tonight and have sex. I don’t want to go out and drink and sleep over with you after you telling me you have a huge problem. Like, are we seriously not going to talk about something that is bugging you so much that you’re telling me “it’s too much now”. It’s frustrating me to no end. Maybe some of you can translate his perspective to me because I’m annoyed as hell.

I broke up with the motherfucker